UPDATE: Applicant and Grantor Login

The Grantor and applicant login issue has been resolved. The issue resided with the OBLIX servers which are part authentication process for user logins to the system. In our troubleshooting of the OBLIX authentication issue, our efforts were focused initially on the four 2007 OBLIX instances.

Subsequent investigation revealed the source of the OBLIX issue originated, not in the four OBLIX instances that we were troubleshooting, but rather in the two, 2006 OBLIX servers.

In addition to defeating Grantor logins, the misbehavior of these two servers caused a cascade of issues. Their repeated unsatisfied LDAP requests caused the excessive LDAP loading that, in turn, caused the login issues that caused applicants to experience problems with submissions or any other LDAP related activity.

We suspect that these two OBLIX servers could have been causing excessive LDAP loading for some time. We believe the correction of this issue will lead to much better overall LDAP performance to enhance the user experience from here forward. We apologize for the inconvenience this caused.

Grantor Login resolved

Grantor users are able to successfully login.

Regards,
Grants.gov PMO

Grantor Login problem

Grantor users are experiencing a login problem with all 3 CSPs. Registered grantor users with ORC, USDA, and Employee Express IDs are being directed to the "Register with Grants.gov" page incorrectly. We are working to resolve the issue.

Regards,
Grants.gov PMO

ALERT: Applicant and Grantor

We are experiencing intermittent login issues for both applicants and Grantors. Applicants who are able to login have been able to successfully submit.

For those applicants submitting with closings today please contact the Grants.gov Contact Center 800-518-4726 if you are unable to meet the closing deadline. We will create a customer service ticket number for you but you also need ot contact the Agency PoC listed on the opportunity indicating you were unable to meet the deadline due to system issues.

Adobe Error Message Explanation

We have updated the potential Adobe Error Messages that applicants may receive when submitting. The errors and their meaning are listed on the site:

http://www.grants.gov/assets/AdobeReaderErrorMessages.pdf

CCR Update

The CCR, FedReg, BRS release has been installed this morning and all parts are back up and operating.

CCR was not down during the upgrade this morning although FedReg and BRS did have to take a short outage.

Login Issue Resolved

Agency and applicant logins are restored to normal. The login problems were caused by corrupt configuration files. Configuration files have been restored from the backup.

We apologize for any inconvenience this may have caused.

Login Issue

We presently are experiencing difficulties with all Grants.gov login capabilities.
Agencies and applicants are experiencing 404 and/or 500 errors in response to authentication requests. We have dedicated all resources to the investigation of the problem cause.

We will provide updates until the system is restored to full operation.

CCR Scheduled Outage February 25

SCHEDULED OUTAGE NOTICE: The CCR and FedReg systems will be unavailable while upgrades are applied for our v4.09.1 release Wednesday, February 25, 2009 between 5:30 AM and 7:30 AM EDT.We apologize for the inconvenience.

ALERT: Grantor Login Issue - E-auth USDA Only

The E-authentication USDA Grantor login is currently experiencing an issue. The USDA login only is unavailable. We are working with USDA to resolve the issue. At this time the impact is on Grantor login only. Applicants are not impacted. We will provide you notice once resolved.

Grant Fraud Information

BUSINESS GRANTS: SORTING OUT THE SCAMS: http://askfsb.blogs.fsb.cnn.com/2009/01/29/business-grants-sorting-out-the-scams/

FREQUENT FRAUD: DON'T OPEN YOUR WALLETS TO THESE SCAMS: http://www.individual.com/story.php?story=95644166

Research and Related Senior Key Personnel Form

Grants.gov has identified an issue where users cannot delete a key person on the Research and Related Senior Key Personnel form from the Project Role field. Grants.gov is looking into a resolution for this issue.

Central Contractor Registration (CCR) Scheduled Maintenance

SCHEDULED MAINTENANCE NOTICE: CCR may be unavailable, at times, while the system goes through scheduled system maintenance tonight, Tuesday, February 17, 2009 between 21:00 and 23:59 EST.We apologize for the inconvenience.

Error Message Explanation

This Error Message: "Could Not Invoke Service, Nested Exception"

MEANS

The grant application submission will be automatically reprocessed by the system. Your application package will either be successfully validated or it will be rejected with valid errors. You can periodically check the status of your submission by using the Check Application Status option under the Applicant Login

ALERT: Applicant Submission Errors

The Grants.gov system is currently returning several different error messages to applicants upon submission. We are working to identify the root cause of these issues. We are asking applicants to contact the Grants.gov Contact Center to create a customer service record identifying any errors they may receive.

Examples of Errors:

- Error Message: Could Not Invoke Service..Nested Exception
- “Failed to Update GrantApplication XML’s LOB”
- “Buffer Overflow”
- “REJECTED WITH ERRORS”

Once we know the cause and meaning of these errors we will provide you with further notice.

Filename Attachments - Adobe & System-to-System (S2S)

We are seeing java.lang.Exception: 16 errors when the Adobe and S2S submissions are not following the filename restrictions. Users have used a single quote in the filenames which causes the error. All S2S users MUST follow the guidance in the FAQ which is listed on the Grants.gov site: http://www07.grants.gov/applicants/submit_application_faqs.jsp#12

Are there restrictions on file names for any attachment I include with my application package?

File attachment names longer than approximately 50 characters can cause problems processing packages. Please limit file attachment names. Also, do not use any special characters (example: &,–,*,%,/,#’, - ) or spacing followed by a dash in the file and for word separation, use underscore (example: my _Attached_File.pdf) in naming the attachments. Please note that if these guidelines are not followed, your application may be rejected.

EXAMPLE 1 - Note 'D' causes an error

[2009-02-11 11:17:46,610][FATAL][GrantS2SFacade.submitApplication()1377]-Couldn't completely receive Submission due to unknown Exception. Throwing Exception to WS Implementation
org.dom4j.InvalidXPathException: Invalid XPath expression: //att:FileLocation[@att:href='D'ASSORO+BCRT+2002_FTLUII5NT77KT4IQDKQGDQBAD6.pdf'] Expected: ]
.
.
java.lang.Exception: 16

EXAMPLE 2 - Note 's causes an error
//att:FileLocation[@att:href='cid:GP_Candidate's_Background.pdf]

ALERT: New Grantor Registration Issue

We are currently experiencing an issue with new Grantor registration. When a new Grantor registers they get the error message; “An internal error has occurred”. The Grantor cannot complete registration.

This has no impact on existing Grantor or Applicant login. If you are a new Grantor and need assistance please contact the Grants.gov PMO staff.

Corrected Issue - Erroneous e-mail Notifications

The erroneous e-mail issue has been corrected as part of Build 2008 – 03. We have tested this issue in production and confirm it is no longer occurring. Here is a recap of the issue from a previous blog posting: http://grants-gov.blogspot.com/2008/12/alert-applicant-e-mail-notifications.html

Regards,
Grants.gov PMO

Applicant Login Work Around

We are continuing to experience applicant login issues.

The following work around has been identified until this issue has been resolved. If you are successfully registered with Grants.gov and experiencing an issue logging in please follow these steps which will enable you to successfully log in.

1. Steps:Go to www.grants.gov
2. On the right side of the page, under "Applicants," click on "Applicant Login"
3. Click "Continue to Login"
4. Select "Forgot my Password"
5. Follow the instructions and then "enter your username"
6. You will then receive an email from grants.gov with a temporary password
7. Click on the link in the email and cut and paste the temporary password into the appropriate box
8. Complete your profile
9. After you complete your profile you will be asked to create your new password
10. You can use your old password if you like
11. Once you reset then you can successfully log in
    

If you have any questions please contact the Grants.gov Contact Center at 800-518-4726.

ALERT: Intemittent Applicant Login Issue

We have been getting intermittent reports of applicant login issues. Some applicants have taken 2 – 3 attempts before successfully getting into the system.

We are investigating the source of the login issue. We have discovered an LDAP query that is taking over 300 seconds to complete. This is consuming resources and slowing down logins. While we are investigating the cause of the improper query we have made available additional hardware resources to help improve the speed in the near term.

We do not have an estimated time to correct but will provide updates as we proceed.

Regards,
Grants.gov PMO

Grants.gov Build 2008 - 03

Grants.gov successfully deployed Build 2008 – 03 which increased the number of concurrent users logged into the system to 2,000. This increase will improve performance. The build changed Step 3 of the registration process where Grants.gov now maintains the username and password functionality internally versus utilizing a third party service provider.

The system was unavailable Saturday, February 7, through 12:01 AM Monday, February 9, 2009 at which point the system went live back into production.

There were no major issues with the deployment. There are two outstanding issues which we are currently working and will notify you once resolved. At this time we do not have an estimated time of resolution but we anticipate correcting shortly.

1. Applicant e-mail: This is an intermittent issue which has arisen during testing. When an applicant logs into the system and they go to manage profile, if they update their e-mail address the updated e-mail address is not taking effect. There is no message telling the applicant of this issue therefore they are unaware their e-mail address has not been updated. Any e-mails sent by the system will be sent to the applicant e-mail address that existed before the change.

2. Grantor Token: When Grantors register they receive an email message with an Activation Token. To associate another credential with this profile, Grantors need the Activation Token. Users who have been assigned the Manage Synopses role will receive a second email message with a user ID and password for use with the Grants.gov Email XML Synopsis service. This token is currently not working.

This morning we have received reports from our Contact Center that applicants are experiencing issues when they go to log in. We are currently working to identify the cause of this issue and will provide you with more details as they become available.

We appreciate your continued patience as we work towards resolving these open issues. If you have any questions please contact the Grants.gov PMO staff.

Regards,
Grants.gov PMO

Grants.gov will be unavailable from 12:00AM Saturday, February 7, 2009 through 8:00AM Monday, February 9, 2009.  Over this period Grants.gov will be implementing changes to improve overall system performance and update forms under System Build 2008-03.  Once the changes are implemented users should experience faster login times, improved system performance, clearer email notifications, and more choices when using the RSS feed.  When the system is brought back online additional information will be available at the Site Features and Enhancements page of Grants.gov.  Grants.gov will provide notice if there is any change to this implementation.        

UPDATE: Grantor Log in Issue Corrected

Working with the E-authentication office we have corrected the Grantor log in issue. Grantors can now successfully log into the system.

We apologize for any inconvenience this may have caused today. If you have any questions please contact the Grants.gov PMO staff.

Regards,
Grants.gov PMO

Grantor login problem

Grantor users are experiencing a login problem with all 3 CSPs. Registered grantor users with ORC and Employee Express IDs are being directed to the "Register for Grants.gov" page incorrectly. USDA users may experience the inability to login from the USDA site. we have contacted USDA E-Auth as well. All parties are working to resolve this problem as soon as possible.

ALERT: System Scheduled Down Time

Grants.gov is scheduled to deploy system Build 2008 – 03 this weekend. To deploy the build we need to take the Grants.gov system offline. The system will be unavailable Saturday, February 7, 2009 through Sunday, February 8, 2009. The system is scheduled to be brought back online at 11:59 PM EST February 8. If the deployment is ahead of schedule we will bring the system online sooner.

This build will increase the number of concurrent users up to 2,000 which will alleviate existing system slowness.

This build corrects the erroneous e-mail notifications issue. We need to conduct a test in production to formally validate this issue has been resolved.

Once this build is deployed the next priority is to correct the CCR data extract pull. We will provide you with more information as both these issues are resolved.

We appreciate your patience as we have worked towards deploying this build. If you have any questions please contact the Grants.gov PMO staff.

Grantors: USDA E-Auth is down

USDA E-Auth is experiencing a problem. USDA has posted a message on their site regarding the service problem they are experiencing. This service problem only impacts grantors attempting to access Grants.gov. ORC and Employee Express are operating.