S2S Submissions Intermittently Stuck During Processing

All:

Since the April Release, some S2S submissions have been getting stuck during processing requiring manual intervention by the Grants.gov Operations and Maintenance Team to complete the processing. Below is a detailed status update for all of you.

Issue: Some S2S submissions are getting stuck with error "Exception caught during S2SReceiving process:null" 

Cause: During the April Release, Grants.gov implemented a new XML parser based on VTD-XML version 2.10. The cause of the parsing issue is due to a bug in the VTD-XML parser, where VALID XML with namespace declarations (used and not used) found only in form elements (not found in the xml root element) caused a problem during parsing of the XML. The bug has been confirmed by developers of the VTD parser.

Frequency: Intermittent. 

Short-term fix: We are cognizant of all closings. The Grants.gov O&M staff are monitoring all submissions and manually processing stuck submissions on a regular basis. That said, there may be a delay in receiving validation, rejection, or received by agency emails.

Long-term fix: The parser is patched by the owner of the API and will resolve the issue. A fix is currently in  process of being tested.

Notes: This does not effect PDF application package submissions.

We enormously appreciate your patience and will keep you abreast of our progress.

Ed

Conversion from CCR to SAM Moved to End of July

From The General Services Administration:

The General Services Administration (GSA) is moving the implementation date of the System for Award Management (SAM) from May 29, 2012 to the end of July 2012. The additional sixty days will allow federal agencies to continue preparing their staff, give agencies and commercial system providers even more time to test their data transfer connections, and will ensure SAM contains the critical, documented capabilities users need from the system. 

This first phase of SAM will include the capabilities of Central Contractor Registration (CCR)/Federal Agency Registration (FedReg), Online Representations and Certifications Application (ORCA), and the Excluded Parties List System (EPLS). In preparation for the launch, GSA conducted extensive testing internally and in coordination with federal agencies using the data from these systems in their own contracting, grants, finance, and other departments. The testing was very valuable and will focus the efforts of the next sixty days. 

SAM will reduce the burden on those seeking to do business with the government. Vendors will be able to log into one system to manage their entity information in one record, with one expiration date, through one streamlined business process. Federal agencies will be able to look in one place for entity pre-award information. Everyone will have fewer passwords to remember and see the benefits of data reuse as information is entered into SAM once and reused throughout the system. 

Please see http://sam.gov for more information about the system.

The Grants.gov program management office strongly encourages all new grant applicants to register with CCR now to ensure a trouble-free and timely submission process during this transition period. The registration process may take between 3-5 days to complete.


If you are already registered with CCR, please make sure that your organization's account information is up-to-date and that your account is not expired (CCR accounts require annual renewals.) To check your CCR account, please visit http://www.ccr.gov.

CCR is going away May 29, 2012. Welcome to SAM.

All:

The Central Contracting Registry (CCR) will be replaced by the System for Award Management (SAM). SAM will be live and active May 29, 2012. (If you go to CCR.gov on May 29 you will be re-directed to SAM.gov.)

SAM is where you will go to register as an institution, organization, or business in order to apply for Federal Assistance (grants) or for contracts as of May 29, 2012.

Active CCR registration is a pre-requisite to the successful submission of grant applications. Some key notes to our grants community:

• If you have a current updated active registration, the process will be seamless to you.
• You will be unable to do a NEW registration with CCR from May 23 - 28, 2012. Any new registrations done during this period will not be processed by SAM until starting May 29, 2012.
• If your registration is set to expire anywhere between the period of May 15 - July 15, 2012, you are automatically granted an additional 90-day extension to renew your registration.
• All existing CCR data will be exported to SAM.

To learn more about the switch from CCR to SAM, you can get more information here. To learn more about SAM, you can go to SAM.gov.

The Grants.gov PMO strongly suggests that you go to CCR prior to May 23, 2012, and check your account. A quick to-do list:

• When does my account expire? 
• Do I need to do my annual renewal of registration?
• Who is my eBIZ POC? Is this person still in my organization? 
• Do I need to update anything?

Please feel free to contact the support desks at CCR, SAM, or Grants.gov with further questions.

S2S Issues Resolved

All:

A fix has been tested and deployed to resolve the remaining S2S issues:

Exception caught during S2SReceiving process:Grant Application XML hash is incorrect.

and

Error:Exception caught during S2SReceiving process:Attachment hash values in the Grant Application XML are out of sync with the attachment(s) sent/received.

Should any of the errors encountered this past week (or new errors) resurface, we ask that you contact the Grants.gov Contact Center and additionally also post to this S2S listserv.

We are cognizant of major closings occurring over the next few days and will continue monitoring the system and this listserv.

We greatly appreciate your patience.

Update on S2S Submissions - XML Hash Error and process:Attachment Hash Values Error

All:

If you experience the following error:

Exception caught during S2SReceiving process:Grant Application XML hash is incorrect.

Please contact our Contact Center immediately at 1-800-518-4726 or send an email to support@grants.gov stating that error issue. In the interim until we come up with a permanent fix, we are able to manually process submissions with this error type. 

(Please note: that we will not process submissions with "typical" business validation rejection messages.)

This is only a temporary work around until we are able to implement a permanent fix.

We are very close to coming up with an interim solution for the following error:

Error:Exception caught during S2SReceiving process:Attachment hash values in the Grant Application XML are out of sync with the attachment(s) sent/received 

We are developing instructions for you and your technical support to also implement these interim solutions.

Ed

Update on S2S Applicant Submission Issues

Summary

We performed released a system build the weekend of April 28-29, 2012. 

As of Monday, April 30, 2012, many in the S2S community started to experience the following issues which continue to today:

Error Message 1

1. XML validation error: Exception caught getting schemaLocation value: null

2. Exception caught during S2SReceiving process:Schema validation failed

or

Error Message 2

1. XML validation error: Exception caught getting schemaLocation value: -1  

2. Exception caught during S2SReceiving process:Schema validation failed

or

Error Message 3

Rejected with Errors:Exception caught during S2SReceiving process:Number of attachments declared in the Grant Application XML does not match the number of attachments sent/received.

or

Error Message 4

Exception caught during S2SReceiving process:Attachment hash values in the Grant Application XML are out of sync with the attachment(s) sent/received.

or

Error Message 5

Exception caught during S2SReceiving process:Grant Application XML hash is incorrect.

What Changed After This Weekend?

In a nutshell, without getting into details and minutiae right now, we've been bringing our system up to par with best practices in XML standards. This adversely impacted you because your systems were developed to old XML practices and standards as instructed in guidance dating back to the old days of Northrop Grumman.

What's Causing these Errors?

About Error Messages 1 & 2 - schemaLocation

These are caused by the same issue. In the XML schema header, the xsi:schemaLocation is missing from your submissions. Example: 

The line highlighted in Orange is missing in your submissions.

http://apply.grants.gov/system/MetaGrantApplication" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:header="http://apply.grants.gov/system/Header-V1.0" xmlns:glob="http://apply.grants.gov/system/Global-V1.0" xmlns:PerformanceSite_1_2="http://apply.grants.gov/forms/PerformanceSite_1_2-V1.2" xmlns:globLib="http://apply.grants.gov/system/GlobalLibrary-V2.0" xmlns:RR_Budget="http://apply.grants.gov/forms/RR_Budget-V1.1" xmlns:att="http://apply.grants.gov/system/Attachments-V1.0" xmlns:RR_OtherProjectInfo_1_2="http://apply.grants.gov/forms/RR_OtherProjectInfo_1_2-V1.2" xmlns:RR_SF424_1_2="http://apply.grants.gov/forms/RR_SF424_1_2-V1.2"xsi:schemaLocation="http://apply07.grants.gov/apply/opportunities/schemas/applicant/oppDE-FOA-0000646-cfda81.049-cidDE-FOA-0000646.xsd">

About Error 3 & 4 - Number of Attachments

This is currently under investigation.

About Error 5 - Hash Values

There was a condition that existed before that did not verify the hash value of submitted applications. This was fixed in the recent build. Please make sure that your application hash value is correct and resubmit.

Next Steps

Our team has been working day and night all week to analyze, address, and bring these issues to resolution. We have developed and tested a fix that will address the schemaLocation issue as this is the prevalent issue. We are rolling back to the way things were so that you do not need to include schemaLocation at this time. (We ask that in the meanwhile you start planning to add this to your XML submission headers. We will release proper documentation far ahead of time before deploying this and will provide a window of time for you to make the necessary adjustments to your systems.)

We are in process of deploying the schemaLocation fix to Production and will inform the community once this has been completed today.

The Number of Attachments issue is still undergoing analysis and will need another day to develop, test, and deploy a fix. We will continue to keep you abreast of resolution to this issue.

As mentioned above regarding Hash Values, please make sure that your application hash value is correct and resubmit.

We recommend that you contact the Program Point of Contact on the funding opportunity announcements for guidance. The PMO has reached out extensively to our Federal agencies to inform them of the issues currently faced by you, the applicants.  Please keep all of your documentation, submission receipts, contact center help desk tickets. 

You always have the alternative to download the Adobe PDF application package from the Grants.gov website to fill out and submit. (Please note that you must be properly registered with DUNS, CCR, Grants.gov and approved as an AOR by your organization's eBIZ Point of Contact in order to successfully submit your application.)

We will continue to keep you posted on further developments.

We thank you enormously for your patience during this time.

Sincerely,

Ed, Kevin, and the entire Grants.gov Team.

Sporadic Issues with S2S Applicant Submissions

Since Monday, April 30, 2012, it was observed that a number of S2S submissions by applicants have encountered  the same error:

Error:Exception caught during S2SReceiving process:Schema validation failed XML validation error: Exception caught getting schemaLocation value: -1

The error encountered by applicants has been traced to a tightening up of schema validation rules (which was done with the recent System Build over the weekend of April 28^th) in order to bring Grants.gov up to par with best practices of XML.

Our team is in process of developing and testing a fix.

The Federal agencies have all been notified of the situation.

If you are encountering this issue, we ask that you contact the program point of contact on your full funding opportunity announcement for guidance.

Please note that this does not effect Adobe PDF submissions.

We will continue to provide updates throughout the day.

Thank you for your patience.