ALERT: Grants.gov February 6 – 9, 2010 Maintenance Outage

In order to bring a major upgrade to the Grants.gov system (referred to as the “boost”) into production, it is necessary to bring Grants.gov off-line from 00:01 AM ET on February 6, 2010 until 11:59 PM on February 9, 2010. The entire Grants.gov system will be unavailable during that time.

The upgrade will further increase the capacity of Grants.gov, along with the system’s reliability and its ability to sustain continuous high volume activity. We have done our best to minimize the length of the maintenance outage necessary to bring these improvements into production.

We are working closely with agencies that previously posted closing deadlines during the outage period to minimize any impact on the agencies or their applicants. Agencies with previously posted closing deadlines on February 6 – 9, 2010 are taking appropriate steps to ensure applicants are able to apply for these posted opportunities. Please note the outage period, and any agency instructions for particular grant opportunities, and plan accordingly when applying for grants in the early February time period.

We apologize for any inconvenience this outage period may cause.

Regards,
Grants.gov PMO

ALERT: Grants.gov January 9, 2010 Maintenance Outage

It is necessary to bring Grants.gov off-line for fifty nine minutes January 9, 2010 from 00:01 until 1:00 AM ET to implement the forms production build. The entire Grants.gov system will be unavailable during this time.

This build will implement fourteen new forms and update thirty-nine forms that are associated with SF 424 Family. If you have any questions or concerns please contact the Grants.gov PMO staff.

Regards,
Grants.gov PMO

Implementing a Maintenance process for the Testing Environment (AT07)

We are implementing a maintenance process for the Testing Environment (AT07) where we will be taking AT07 offline weekly on Monday and Thursday evenings from midnight through 2:00 AM EST.

This regular maintenance will ensure AT07 (our testing environment only)has sufficient memory to meet the needs of our customer base. No enhancements will occur in this process.



Regards,
Grants.gov PMO

Grants.gov Fiscal Year 2009 Annual Report

Fiscal Year (FY) 2009 was a year of great growth for Grants.gov. In FY 2009 Grants.gov processed 309,771 applications which was a 53% increase over the 202,133 applications processed in FY 2008.

There were 64,409 applications processed for Recovery Act specific opportunities posted on the site. This represents 21% of the total applications processed in FY 2009. For more detailed information on FY 2009 please see the Grants.gov FY2009 Annual Report:

-- http://www.grants.gov/assets/AnnualReport2009.pdf

Regards,
Grants.gov PMO

Grants.gov System Unavailable November 14

The Grants.gov system will be unavailable Saturday, November 14, 2009, from 12:01 a.m. EST through 6:00 a.m. EST. The system will be unavailable due to activities pertaining to the Technical Augmentation Design (Boost) effort. If you have any questions or issues regarding the system availability, please contact the Help Desk.

Regards,
Grants.gov PMO

Grants.gov New Contact Center Hours

Beginning today, October 19, 2009, the Grants.gov Contact Center will be available 24 hours a day, 7 days a week. The Contact Center provides customer service to the applicant community. The new extended hours will provide applicants support around the clock, ensuring the best possible customer service is received any time it’s needed. You can reach the Grants.gov Support Center at 1-800-518-4726 or by email at support@grants.gov.

ALERT: Grants.gov System Issue August 10, 2009

Grants.gov experienced a database issue on August 10, 2009 that impacted both Find and Apply functionality. Beginning at approximately 2:30 PM EDT applicants began experiencing intermittent issues searching for opportunities and applying as a result of slow system performance. This slow performance caused some interruptions in service when applicants were submitting applications requiring them to submit again. The issue was intermittent and did not impact all applicants.

The Apply functionality was unavailable from 8:05 PM and the system experienced a database crash at approximately 9:30 PM and applicants were unable to submit applications until the database was brought back online at 11:12 PM.

We are continuing to investigate the root cause of the issue and will provide you with more information once it becomes available.

The find and apply functionality is currently operating. We apologize for the inconvenience this may have caused.

Regards,
Grants.gov

August Build Successfully Deployed

The implementation of Build 2009-04 for the Grants.gov system was successfully completed on August 9. Learn more about the system enhancements on the Site Features and Enhancements page: http://www.grants.gov/aboutgrants/buildreleases.jsp

Regards,
Grants.gov

Grants.gov System Unavailable August 8 - 9

Due to the implementation of Build 2009-04 the Grants.gov system and website will be unavailable Saturday, August 8, 2009 through Sunday, August 9, 2009. The system is scheduled to be back online at 11:59 p.m. EDT August 9, 2009. The system updates for this build will be available once the build is successfully deployed on the Site Features and Enhancements page: http://www.grants.gov/aboutgrants/buildreleases.jsp.

We appreciate your patience. If you have any questions or issues during the system downtime, please email support@grants.gov and the Contact Center will address your inquiry during business hours (Monday-Friday, 7:00 a.m. to 9:00 p.m., EDT).

Regards,
Grants.gov

Recovery Act and Grants.gov

A new feature was unveiled on Grants.gov to help users find and apply for The American Recovery and Reinvestment Act (ARRA) grant opportunities.

The launch of the ARRA or Recovery Act feature on the homepage of Grants.gov will direct users to Recovery Act opportunities, other Recovery Act resources, upcoming Webinars and links to www.whitehouse.gov/recovery and www.recovery.gov.

Regards,
Grants.gov

Grants.gov - Internet Browser Compatibility

Grants.gov routinely tests compatibility with numerous browser and operating system versions and combinations. While testing a newer version of Firefox and the beta version of Chrome, we identified potential compatibility issues with the current version of Firefox. We reported that compatibility issue to you prematurely. Further testing shows that the earlier compatibility issues were unfounded. As a result, we need to retract our earlier message regarding browser incompatibilities with our apologies for any confusion or concern related to this miscommunication.

Firefox and Internet Explorer browsers on Windows XP and Vista, and Firefox and Safari on Mac systems are compatible with Grants.gov. Also, based on our limited testing there were no issues found with using Google Chrome on Windows XP and Vista. Please note that Chrome is considered Beta software by Google (http://www.google.com/chrome/intl/en/why.html).

Again, we sincerely apologize for any confusion our previous communication may have caused. If you have any questions please contact the Grants.gov Contact Center (1-800-518-4726) or via e-mail support@grants.gov.

Regards,
Grants.gov PMO

Grants.gov Webcast

Please join us July 15th from 1:00 - 2:00 PM EDT for our Quarterly Grants.gov Stakeholder Webcast: http://grants.gov/help/stakeholder_communications.jsp

Agenda
- Grants.gov Activity Update  
- Grants.gov Recently Accomplished System Builds
- Grants.gov “Boost”   
- Applicant System-to-System (S2S)

ALERT: Grants.gov E-mail Subscriptions

We are currently experiencing an issue where people are unable to sign-up for daily e-mail notifications of new grant opportunity postings. Both the “All Grant Notices” and “Notices Based on Advance Criteria” are currently unavailable to subscribe too.

Until the issue is resolved please call the Contact Center (1-800-518-4726) with the subscription criteria you would like to receive and we will manually enter you in the Grants.gov system.

Please note; you can utilize the RSS feed to be notified of new opportunity postings by Agency and category. We apologize for the inconvenience this may cause.

Regards,
Grants.gov

June Build Successfully Deployed

The June build was successfully deployed on June 27 - 28. There are no issues at this time. The build communication detailing the system enhancements is on the site features and enhancements page: http://grants.gov/aboutgrants/buildreleases.jsp

Regards,
Grants.gov PMO

Grants.gov June 27 - 28 Build 2009-03

The June build has been successfully tested and will be deployed into production June 27 - 28. The system will be unavailable June 27 – 28 to implement the build.

The detailed build communication outlining the benefits will be available once the system is back online on the site enhancements page: http://grants.gov/aboutgrants/buildreleases.jsp

Regards,
Grants.gov PMO

Grants.gov June 27 - 28 Build 2009-03

The next system build is scheduled to occur June 27 – 28. The system will be unavailable during this time. We will distribute the detailed build communication next week once the build has been tested and officially authorized for deployment into production.

Benefits of June Build 2009-03

Streamlined Applicant Registration
– New applicant registrant username and password portion of the Grants.gov profile have been streamlined. Originally there were three screens to complete the process, there will now be two.

Improved Email Notification (Electronic Business Point of Contact (E-Biz POC) Email)
– Email notification sent to the Grants.gov usage authorizing official within your organization (E-Biz POC) will now contain the contact information of the AOR so the E-Biz POC will know immediately who they are considering for approval.

Immediate Status Updates (Authorized Organization Representative (AOR) Status)
– Applicants will immediately be able to see their AOR status upon login to the Applicant Center.

Clearer Messaging (Track My Application)
– The track my application page will have added functionality and return clearer messages along with the status of the Grants.gov tracking number.

Direct Access to Applicant Login
– Applicants will now have direct access to the Applicant Login link available from any page of the website in the Navigation.

Detailed information will be available on these system enhancements on the Site Features and Enhancements page once the build is deployed.

Regards,
Grants.gov PMO

Adobe Reader Versions 8.1.6 & 9.1.2 Released

Adobe has released Adobe Reader versions 8.1.6 and 9.1.2. Grants.gov has successfully tested these versions and they are compatible. We are in the process of updating the website to reflect these latest compatible versions.

Regards,
Grants.gov PMO

UPDATE: Applicant Submission Issue

Here is a recap of the issue we experienced on Friday, June 5, 2009.

Date/Time: June 5, 2009 2:06 – 2:56 pm EDT

Issue: Submissions were not being accepted into the system as a result of a shortage of free space on one of the database drives. The database archive logs reached the allotted space capacity and caused the database to become unavailable.

Impact: Applicants were unable to submit applications.

Resolution: Database system maintenance returned sufficient space to the system to allow submissions to be accepted. The database archive log scripts have been adjusted to free up more database space.

Follow-up Actions: The Technical Augmentation Design (Boost) will add additional database space. Until the Boost is implemented Database maintenance procedures are being reviewed, modified, and followed to prevent a recurrence of the issue.

Regards,
Grants.gov PMO

RESOLVED: Applicant Submission Issue

The issue has been resolved. All submissions are now being received. We apologize for any inconvenience this may have caused.

Regards,
Grants.gov PMO

ALERT: Application Submission Issue

We are currently experiencing an issue with the receipt of applications. We are working to resolve the issue and will provide you with notice once resolved.

If you have a closing deadline today and continue to experience an issue submitting please call the Contact Center so your issue is recorded; 1-800-518-4726.

We apologize for the inconvenience.

Scheduled System Maintenance May 30 - 31

Grants.gov will be unavailable Saturday and Sunday, May 30-31, 2009 for system maintenance. We apologize for any inconvenience.

Over 200,000 Submissions Received!

Grants.gov has reached a new milestone, exceeding the number of submissions in FY 08 before the end of FY 09, as a result of the Recovery Act grant opportunities. On May 18, 2009, Grants.gov processed a total of 205,591 applications, surpassing the 202,133 applications processed in FY 08 with more than four months remaining in FY09.

The unprecedented number of applications processed so early in the fiscal year illustrates the success of the initiative. Grants.gov supports electronic applications that can be downloaded to any computer and provides online user support tools and personalized assistance from a dedicated customer support team. The result is a simplified and effective environment for the grant community to find and apply for grants.

Adobe Reader versions 8.1.5 and 9.1.1 released and compatible with Grants.gov

Adobe has released Adobe Reader versions 8.1.5 and 9.1.1. Grants.gov has tested both these versions and they are compatible. We are in the process of updating the website to reflect these new versions. The update will include adding these versions to the existing compatible version table: http://www07.grants.gov/help/download_software.jsp#adobe811

Check Your Grants.gov Registration Status

To check your registration status, login to Grants.gov with the username and password you created during step three of the registration process. Once logged in, click “Manage Applicant Profile”, if your registration is complete and approved you will see the message “AOR Status: Approved”.

About Registration
Grants.gov registration is a one time process. However, there is one step that requires yearly renewal (step two – CCR, please see Organization Registration page for details) if your organization doesn’t renew their CCR registration this will cause delays when attempting to submit a grant application package.

Registration is required in order to submit grant application packages. If you belong to an organization, you must be designated an Authorized Organization Representative (AOR) in order to submit grants on behalf of your organization. The E-Business Point of Contact (E-Biz POC) within your organization must approve your registration by logging into Grants.gov. You should receive an email once your registration is approved by your organization’s grant administrator (E-Biz POC).

The Grants.gov team is pleased to announce that the backlog of applications waiting to be validated by Grants.gov has been cleared. We expect that validations will be able to keep up with submissions going forward.

Grants.gov has focused significant attention and resources over the last several weeks toward being able to successfully receive a record number of applications leading up to a major grant closing on April 27. Grants.gov successfully received 30,000 applications over an 8-day period from April 20 – 27, nearly as many applications as we received in all of March. In focusing on receipts, however, we realized that we would then be in a position where Grants.gov would be able to receive applications much faster than it could validate them. We felt it was less disruptive to have slower validations of applications than for applicants to have problems submitting applications in the first place.

As a result of the record number of applications received, the backlog of applications to be validated rapidly increased, especially for those submitted directly to Grants.gov rather than through system-to-system submissions. This past week, many applications have taken five days or more to clear validations and be ready for agency download and further evaluation. Our target service level is to validate applications within 48 hours of receipt.

Grants.gov implemented a system change on Friday, May 1, that has allowed us to clear the backlog of applications awaiting validation. Equally important, we expect this change to minimize future backlogs of application validations as we achieve a better balance between receipt and validation processing speeds.

We understand the inconvenience and concern that the slower-than-normal application validations have caused, whether to grantor review processes or to applicants who have been looking in vain for their validation messages. We appreciate your patience as we have worked literally night and day to meet the challenges of unprecedented processing volumes.

The Grants.gov team continues to implement its "boost" intiative and other changes to ensure stable operations at continued high volumes and to improve the user experience for both grantor agencies and grant applicants. We look forward to continued and improved dialogue with users to ensure that Grants.gov fulfills expectations as the Federal Government's primary resource for finding and applying for grant opportunities.

AT07 Unavailable Evening April 30, 2009

Grants.gov will be conducting testing on AT07 from 5:30 PM EDT, April 30, 2009 through 12:00 AM EDT, May 1, 2009. Users are asked to not use AT07 during this period. Thank you for your cooperation.”

Grants.gov Submissions

Last evening, April 27, between 7:45 and 9:45 p.m. EDT submissions were not being processed through the system as a result of the SAN device capacity limitation. The SAN device is used for storing queued submissions, the device ran into an upper limit on UFS Solaris Operating System and halted submission acceptance. The system was unable to create new directories to accept incoming submissions. Corrective action has been taken.

From April 20 – 27 we processed over 28,000 applications. In this eight day period alone we surpassed the total monthly submissions for each month in FY 09, with the exception of March, when we processed 36,253 submissions. On April 27 we processed 8,392 applications, the largest one day submission total, surpassing the April 24 record high of 5,973.

On April 27 the Contact Center received 2,065 calls which was the third highest volume of calls we have received to date.

Form in Form submissions are taking greater then 48 hours to validate. We are working to validate all submissions within the 48 hour validation process.

Regards,
Grants.gov PMO

Two Hour Gap in Accepting Submissions on Monday Evening

At approximately 7:45pm, April 27th, the portion of the Grants.Gov system that manages the intake of new applications suffered technical difficulties due to the unprecedented number of new applications received today.

The immediate result was that users could not submit grant applications through Grants.gov for about a two-hour period from 7:45 pm to 9:45 pm EST. Grant opportunity search capability, and processing of applications that had already been submitted before the failure, continued to function normally.

Users who were not able to submit their applications should try submitting again now that full functionality has been restored.

Grants.Gov Program Management Office

Intermittent Issues with Password Reset and Registration

There are currently intermittent issues with resetting a password and registration. We are currently working to resolve these intermittent issues. We apologize for any inconvenience this may cause you. If you experience an issue with resetting your password or registration do not submit multiple requests, please call the Grants.gov Contact Center at 1-800-518-4726 or email support@grants.gov.

Expect Delays in Receiving the Application Submission Validation Email

There is currently a delay with the validation email that Grants.gov sends due to the unprecedented volume of application submissions. The validation email may take longer than Grants.gov’s standard 48 hours.

We appreciate your patience and encourage you to visit the Track My Application page (https://apply07.grants.gov/apply/checkApplStatus.faces) to check the status of your application submission. If you need further assistance please call the Grants.gov Contact Center at 1-800-518-4726 or email support@grants.gov.

Expect Delays in Receiving the Application Submission Validation Email

There is currently a delay with the validation email that Grants.gov sends due to the unprecedented volume of application submissions. The validation email may take longer than Grants.gov’s standard 48 hours.

We appreciate your patience and encourage you to visit the Track My Application page (https://apply07.grants.gov/apply/checkApplStatus.faces) to check the status of your application submission. If you need further assistance please call the Grants.gov Contact Center at 1-800-518-4726 or email support@grants.gov.

Intermittent Issues with Password Reset and Registration

There are currently intermittent issues with resetting a password and registration. We are currently working to resolve these intermittent issues. We apologize for any inconvenience this may cause you. If you experience an issue with resetting your password or registration do not submit multiple requests, please call the Grants.gov Contact Center at 1-800-518-4726 or email support@grants.gov.

April 27, 2009: High Submission Volume

There are 29 Grant Opportunities closing on Monday, April 27, 2009, including a large Recovery Act opportunity for NIH that is expected to receive an unprecedented number of applications. The submissions for these opportunities have already begun and will continue to grow as we move towards April 27. The following agencies have closings on this date; ACF, AMS, CDC, CSREE, DOE, EPA, HRSA, NIH and USAID.

Please remember to click save and submit only one time to begin the submission process. Multiple clicks on Save and Submit will not help. This will only cause an additional resource drain on the system.

After receiving your confirmation page and/or submission receipt email, it may take up to 48 hours for Grants.gov to process your application, send the submission validation email and make the application available to the grantor agency. Please be patient and allow your submissions to work through the system.

The peak hours for submission are 2:00 – 6:00 PM EDT. If you can submit outside this timeframe that will increase the speed at which you can submit.

If you need assistance please call the Grants.gov Contact Center 1-800-518-4726. Right now the wait time for your call is 31 seconds.

Grants.gov April 18 - 19 Build Successfully Deployed

The build was successfully deployed. We have made configuration changes to the system as well as implemented new functionality for applicants to track up to five Grant Tracking #s without having to login to the system.

For more detailed information on the new functionality the build communication is on the site: http://www07.grants.gov/assets/TrackApplicationEnhancement2009-SCR8719.pdf

Regards,
Grants.gov PMO

Grants.gov Scheduled Maintenance April 18 - 19

We are conducting the scheduled maintenance on April 18 - 19 in preparation for several large closings on April 27. On that day there are 27 closings including an NIH closing where they expect a high volume of submissions.

Our efforts this weekend are to improve system performance to meet this demand. By introducing new "Track my Application" functionality this will reduce the number of people logged into the system which will improve performance. Once this functionality is in place users can track up to five Grant Tracking #s without logging into the system.

Moving forward we are executing monthly builds to continue to enhance the system. The next build is tentatively scheduled for May 30 - 31. Once we finalize these dates and the dates for the upcoming builds after that we will notify the community.

Stakeholder Webcast April 15

The Grants.gov Quarterly Stakeholder Webcast is April 15 from 1:00 p.m. - 2:00 p.m. EDT. To view the webcast click here: http://www.hhs.gov/grants/webcast/index.html at 1:00 p.m. on Wednesday, April 15.

*Please note you will need the Adobe Flash Player (http://www.adobe.com) installed on your computer in order to view the live video stream. You can test your ability to view the flash video stream starting at 10:00 a.m. EDT on the day of the webcast at the address listed above. If you are unable to view this live stream in the flash format, it will be available in flash video-on-demand shortly after the broadcast.

Grants.gov Scheduled System Maintenance

The Grants.gov site will be unavailable April 18 - 19 to conduct system maintenance. As part of this effort we will be introducing new functionality to "Track my Application" without login to the system. Once the system comes back online applicants will be able to enter up to 5 Grant Tracking #s to get a status without login to the system.

Upon submitting an application the receipt confirmation page will have a link at the bottom of the page for you to click and check the status of the Grant Tracking # on the Receipt Confirmation page. The receipt e-mail itself will also have a link in it to check the status of that Grant Tracking # as well.

For those of you who utilize AT07 the site will be intermittently offline this afternoon beginning at 1:00 PM EDT through COB today.

Regards,
Grants.gov PMO

Grants.gov Submission Volume

Grants.gov processed 38,646 applications in March 2009 which was a 78% increase over the 21,651 applications processed in February.

For April 1 – 7 we processed 11,464 which is the largest weekly submission total to date for the 2007 Adobe system. April 6 was the largest one day total processed; 3,555.

In our efforts to continue to enhance system performance we are on target with the next build which will occur April 18 – 19. The system will be unavailable during this time. We are changing the system configuration to prioritize login requests. We are also adding the Track my application feature where applicants can track their application status without login to the system. This will further reduce the number of people who login to the system.

More information including the build communication will follow.

Regards,
Grants.gov PMO

NO Testing Allowed in Production

For our System-to-System (S2S) community, any testing in Production must be stopped immediately. There is no testing allowed in our Production environment.

Please use AT07 for all your testing.

Grants.gov PMO

Contact Center Call Volume

In December 2008 the Grants.gov Contact Center received 10,102 calls. In March 2009 the Center received 36,287 for a 259% increase over December 2008. This resulted in increased customer wait times.

Today, April 6, we have 77 Customer Support Representatives (CSRs) available to take calls. Today, we have started a new hire class of 25 Customer Support Representatives. When all 25 complete training we will have 102 CSRs on the phone April 20 to assist customers.

We appreciate your patience as we staff up to meet the unprecedented call volume.

If you need assistance; Grants.gov Contact Center 1-800-518-4726.

CORRECTED: Applicant Submission Issue

The issue has been resolved. Applicants are now able to submit without experiencing any issues. Last evening in our efforts to correct the 500 Error message applicants and Grantors were intermittently receiving these efforts triggered the slow submission time for applicants overnight and this morning.

Applicants, please note, if you cannot submit you should contact the Grants.gov Contact Center (800-518-4726) to create a customer service ticket. By trying to submit multiple times it creates an additional resource drain on the system further slowing down processing times. We appreciate your consideration in this matter.

ALERT: Application Processing

Applicants are currently experiencing issues submitting applications. Applications are getting processed but some applicants are having issues uploading. We will provide you with more information as it becomes available. We
appreciate your patience.

If you have a grant deadline today please contact the Grants.gov Contact Center 1-800-518-4726. The Center Staff starts in staggered shifts and will be fully staffed by 9:00 a.m.

Regards,

Grants.gov PMO

Application Upload Connectivity

Users may experience connectivity problems with Grants.gov during application submission. We are aware of the situation and working on a resolution.

Regards,
Grants.gov PMO

Grants.gov Applications Processed

Grants.gov has experienced and unprecedented volume on the website. The hits on the site have increased 115% from December 08 (800K) to March 09 (1.8M).

In the three week period from March 2 - 22 Grants.gov processed 27,576 applications which is by far and away the largest submissions processed in any 3 week period since Grants.gov began.

We are continuing to work towards enhancing system performance due to the increased activity on the system.

Recovery Act Opportunities

On March 6, 2009 Grants.gov began capturing Recovery Act specific grant opportunities posted by Agencies. We created the following page http://www.grants.gov/applicants/recovery.jsp with information on the Act as well as access to the opportunities.

From March 6 through March 29 this page has been accessed 94,227 times.

Grants.gov Build 2009 - 01 Successfully Deployed

Grants.gov successfully deployed Build 2009 – 01 on March 29, 2009. This build impacted Grantors only. At this time we have no issues reported. If you are experiencing any issues please contact the Grants.gov PMO staff.

If a Grantor is experiencing an issue with the username and/or password please remember from last week’s discussions there is a work around the Super User can execute. The Super User can delete the Grantor and then have them re-register.

Regards,
Grants.gov PMO

Grants.gov System Downtime

Due to the implementation of Build 2009-01 the Grants.gov system and website will be unavailable Saturday, March 28, 2009 through Sunday, March 29, 2009. The system is scheduled to be back online at 11:59 p.m. EDT March 29, 2009. This build only impacts Grantors, there is no impact to applicants.

We appreciate your patience. If you have any questions or issues during the system downtime, please contact support@grants.gov the Contact Center will address your inquiry during business hours (Monday-Friday, 7:00 a.m. to 9:00 p.m., EST).


Regards,
Grants.gov

UPDATE: Grantor Login Only

The USDA E-Auth portal is now functioning. Grantors should have no issues logging in.

This issue had no impact on applicant login.

Grantor Login Only

The USDA E-Auth portal is experiencing system difficulties. This impacts Grantors only. Applicants are able to successfully login.

The issues experienced yesterday have been resolved. This USDA E-Auth issue is separate from yesterday’s system issues.

RESOLVED: Limited System Availability Monday, 3/16/09, Starting at 8am

UPDATE:

Full functionality has been restored to Grants.Gov as of 4:10 pm on 3/16/09. Grant Search, Grant Application Download, and Grant Application Submission are now operational. We apologize for any inconvenience.

======================================
THE ISSUE DESCRIBED BELOW HAS NOW BEEN RESOLVED:
Grants.Gov is offering limited functionality at this time related to a database server issue we experienced this morning. The server went offline at approximately 8am, 3/16/09, and will not be returned to service until it has been quality checked by the Grants.Gov System Integrator. The functional areas affected by this database server issue include Grant Search, Grant Application Download, and Grant Application Submission. There may also be other functions affected by this issue.

The Grants.Gov Program Management Office will keep our user community informed by updating this blog as information on system status becomes available.

We do not know the specific time that full functionality will be restored. Currently we expect Grants.Gov to be restored by Monday evening, 3/16/09.

Central Contractor Registration Issue Resolved

The CCR data extract issue has been resolved. All CCR data has been updated into the Grants.gov system and is current. Tonight we will begin the daily data extract which keeps the CCR information up to date in Grants.gov.

The issue at hand was that Grants.gov was unable to pull updated or new CCR registrants from the CCR database. All CCR registrants who registered or modified their information since December 31, 2008 have now been updated into the Grants.gov database therefore the applicant login is now active.

High User Volume continues

Grants.gov continues to experience system slowness due to the high volume of users. We have over 2500 users logged-on and over 1200 users conducting searches. We will continue to provide updates.

Regards,
Grants.gov PMO

Experiencing System Slowness

At the present time the Grants.gov system is experiencing slowness due to the volume of users. We have 3000+ persons logged-on at the moment and 1000+ conducting searches. We will provide continuing updates as the day continues.

Regards,
Grants.gov PMO

ALERT: E-business Login

We are currently experiencing an intermittent issue with the E-business login. When some applicants go to login they are getting either a blank page or an "Error 403--Forbidden" message.

If you encounter an issue with the E-business login please contact the Contact Center (800-518-4726) and we will work to resolve your issue.

Applicant System-to-System (S2S)

NOTICE: Applicant System-to-System Endpoint Access Limitation

The system is currently experiencing strain due to frequent user access. Applicant System-to-System users are asked to limit their entry into the system for querying opportunities and checking application submission status to NO MORE THAN ONCE PER HOUR to reduce the load on the LDAP server.

We appreciate your consideration.

Contact Center - Phone Line

Due to the increased call volume on the Grants.gov Contact Center number (1-800-518-4726) the current phone lines are running at capacity which is intermittently causing a busy signal. The vendor has identified additional resources to expand capacity. We will continue to monitor this closely and provide updates as necessary.

If you receive a busy signal when calling in please send an e-mail: www.support@grants.gov and we will respond.

System-to-System Information

All agencies have successfully transitioned from PureEdge to Adobe. This evening, March 5, 2009, we will be turning off the 2006 endpoint for System-to-System (S2S).

TURNING OFF
Agency S2S 2006 system: https://ws.grants.gov:446/agency-s2s-server/services/AgencyIntegrationSoapPort

Applicant S2S 2006 system: https://ws.grants.gov:446/app-s2s-server/services/

If you have not already made the switch you now need to point to the 2007 endpoints only:

2007 ENDPOINTS

Agency S2S 2007 system: https://ws07.grants.gov:446/agency-s2s-server/services/AgencyIntegrationSoapPort

Applicant S2S 2007 system: https://ws07.grants.gov:446/app-s2s-server/services/


Regards,
Grants.gov PMO

Individual Applicants Only

Individuals applying for individual grants only. If you receive the below message or any other error message please contact the Grants.gov Contact Center (800-518-4726) with your Grants.gov Tracking # (If you received one) and we will assist you. In the case of this error message we are able to re-process your application to ensure it is successfully in the system for Agency retrieval. Any other error messages will need to be specifically addressed by our support team.
----------------------------
REASON FOR REJECTION

1. The grant opportunity for which you have applied is designated for individuals submitting on their own behalf. However, your application for a grant/grants was submitted on behalf of a company, organization, institution, or government. An individual submits a grant on their behalf, and not on behalf of a company, organization, institution or government. Individuals sign the grant application and its associated certifications and assurances that are necessary to fulfill the requirements of the application process.

PureEdge to Adobe

The last day to submit a PureEdge application package was Wednesday, March 4, 2009. Grants.gov has moved to Adobe application packages for all agencies.

Enhancements to LDAP Planned. Improved System Responsiveness is the Goal.

Grants.gov is experiencing an extraordinarily high volume of users. The point of entry into the Grants.gov system for applicant log-in is called “the LDAP” - an application protocol for querying directory services. The LDAP is presently configured to handle a peak of 2,000 simultaneous users. Lately, we are confronting upwards to 3,000 simultaneous users by mid-day. When the LDAP is engaged by such a high number of queries, it continues to operate but more slowly. Moreover, accessing the LDAP is extremely difficult. Grants.gov customers are naturally frustrated by these technical challenges.

We understand that entities engaging Grants.gov early in the morning experience minimal difficulty; whereas those entities which wait until later in the day are experiencing significant difficulty.

Our recommendation is two-fold: whenever possible, please do not wait until the last minute of the last day for submissions and please engage the Grants.gov system during its non-peak hours, outside of 11:00 AM EDST to 8:00 PM EDST timeframes to increase your opportunity for a satisfactory experience.

Grants.gov is actively working with the Grants Executive Board on a proposal for increased LDAP capacity.

ALERT: Applicant Login

Applicants are experiencing issues logging into the system. One of the causes of this is the large volume of users on the system. The increase is causing error messages and intermittent inability to log in.

The Grants.gov Contact Center call volume is three times above normal so when you call the Contact Center to report your issue you will experience delayed wait times. We are in the process of bringing 25 additional Custoemer Support Representatives on the phone once training is complete which should alleviate the wait time.

UPDATE: Applicant and Grantor Login

The Grantor and applicant login issue has been resolved. The issue resided with the OBLIX servers which are part authentication process for user logins to the system. In our troubleshooting of the OBLIX authentication issue, our efforts were focused initially on the four 2007 OBLIX instances.

Subsequent investigation revealed the source of the OBLIX issue originated, not in the four OBLIX instances that we were troubleshooting, but rather in the two, 2006 OBLIX servers.

In addition to defeating Grantor logins, the misbehavior of these two servers caused a cascade of issues. Their repeated unsatisfied LDAP requests caused the excessive LDAP loading that, in turn, caused the login issues that caused applicants to experience problems with submissions or any other LDAP related activity.

We suspect that these two OBLIX servers could have been causing excessive LDAP loading for some time. We believe the correction of this issue will lead to much better overall LDAP performance to enhance the user experience from here forward. We apologize for the inconvenience this caused.

Grantor Login resolved

Grantor users are able to successfully login.

Regards,
Grants.gov PMO

Grantor Login problem

Grantor users are experiencing a login problem with all 3 CSPs. Registered grantor users with ORC, USDA, and Employee Express IDs are being directed to the "Register with Grants.gov" page incorrectly. We are working to resolve the issue.

Regards,
Grants.gov PMO

ALERT: Applicant and Grantor

We are experiencing intermittent login issues for both applicants and Grantors. Applicants who are able to login have been able to successfully submit.

For those applicants submitting with closings today please contact the Grants.gov Contact Center 800-518-4726 if you are unable to meet the closing deadline. We will create a customer service ticket number for you but you also need ot contact the Agency PoC listed on the opportunity indicating you were unable to meet the deadline due to system issues.

Adobe Error Message Explanation

We have updated the potential Adobe Error Messages that applicants may receive when submitting. The errors and their meaning are listed on the site:

http://www.grants.gov/assets/AdobeReaderErrorMessages.pdf

CCR Update

The CCR, FedReg, BRS release has been installed this morning and all parts are back up and operating.

CCR was not down during the upgrade this morning although FedReg and BRS did have to take a short outage.

Login Issue Resolved

Agency and applicant logins are restored to normal. The login problems were caused by corrupt configuration files. Configuration files have been restored from the backup.

We apologize for any inconvenience this may have caused.

Login Issue

We presently are experiencing difficulties with all Grants.gov login capabilities.
Agencies and applicants are experiencing 404 and/or 500 errors in response to authentication requests. We have dedicated all resources to the investigation of the problem cause.

We will provide updates until the system is restored to full operation.

CCR Scheduled Outage February 25

SCHEDULED OUTAGE NOTICE: The CCR and FedReg systems will be unavailable while upgrades are applied for our v4.09.1 release Wednesday, February 25, 2009 between 5:30 AM and 7:30 AM EDT.We apologize for the inconvenience.

ALERT: Grantor Login Issue - E-auth USDA Only

The E-authentication USDA Grantor login is currently experiencing an issue. The USDA login only is unavailable. We are working with USDA to resolve the issue. At this time the impact is on Grantor login only. Applicants are not impacted. We will provide you notice once resolved.

Grant Fraud Information

BUSINESS GRANTS: SORTING OUT THE SCAMS: http://askfsb.blogs.fsb.cnn.com/2009/01/29/business-grants-sorting-out-the-scams/

FREQUENT FRAUD: DON'T OPEN YOUR WALLETS TO THESE SCAMS: http://www.individual.com/story.php?story=95644166

Research and Related Senior Key Personnel Form

Grants.gov has identified an issue where users cannot delete a key person on the Research and Related Senior Key Personnel form from the Project Role field. Grants.gov is looking into a resolution for this issue.

Central Contractor Registration (CCR) Scheduled Maintenance

SCHEDULED MAINTENANCE NOTICE: CCR may be unavailable, at times, while the system goes through scheduled system maintenance tonight, Tuesday, February 17, 2009 between 21:00 and 23:59 EST.We apologize for the inconvenience.

Error Message Explanation

This Error Message: "Could Not Invoke Service, Nested Exception"

MEANS

The grant application submission will be automatically reprocessed by the system. Your application package will either be successfully validated or it will be rejected with valid errors. You can periodically check the status of your submission by using the Check Application Status option under the Applicant Login

ALERT: Applicant Submission Errors

The Grants.gov system is currently returning several different error messages to applicants upon submission. We are working to identify the root cause of these issues. We are asking applicants to contact the Grants.gov Contact Center to create a customer service record identifying any errors they may receive.

Examples of Errors:

- Error Message: Could Not Invoke Service..Nested Exception
- “Failed to Update GrantApplication XML’s LOB”
- “Buffer Overflow”
- “REJECTED WITH ERRORS”

Once we know the cause and meaning of these errors we will provide you with further notice.

Filename Attachments - Adobe & System-to-System (S2S)

We are seeing java.lang.Exception: 16 errors when the Adobe and S2S submissions are not following the filename restrictions. Users have used a single quote in the filenames which causes the error. All S2S users MUST follow the guidance in the FAQ which is listed on the Grants.gov site: http://www07.grants.gov/applicants/submit_application_faqs.jsp#12

Are there restrictions on file names for any attachment I include with my application package?

File attachment names longer than approximately 50 characters can cause problems processing packages. Please limit file attachment names. Also, do not use any special characters (example: &,–,*,%,/,#’, - ) or spacing followed by a dash in the file and for word separation, use underscore (example: my _Attached_File.pdf) in naming the attachments. Please note that if these guidelines are not followed, your application may be rejected.

EXAMPLE 1 - Note 'D' causes an error

[2009-02-11 11:17:46,610][FATAL][GrantS2SFacade.submitApplication()1377]-Couldn't completely receive Submission due to unknown Exception. Throwing Exception to WS Implementation
org.dom4j.InvalidXPathException: Invalid XPath expression: //att:FileLocation[@att:href='D'ASSORO+BCRT+2002_FTLUII5NT77KT4IQDKQGDQBAD6.pdf'] Expected: ]
.
.
java.lang.Exception: 16

EXAMPLE 2 - Note 's causes an error
//att:FileLocation[@att:href='cid:GP_Candidate's_Background.pdf]

ALERT: New Grantor Registration Issue

We are currently experiencing an issue with new Grantor registration. When a new Grantor registers they get the error message; “An internal error has occurred”. The Grantor cannot complete registration.

This has no impact on existing Grantor or Applicant login. If you are a new Grantor and need assistance please contact the Grants.gov PMO staff.

Corrected Issue - Erroneous e-mail Notifications

The erroneous e-mail issue has been corrected as part of Build 2008 – 03. We have tested this issue in production and confirm it is no longer occurring. Here is a recap of the issue from a previous blog posting: http://grants-gov.blogspot.com/2008/12/alert-applicant-e-mail-notifications.html

Regards,
Grants.gov PMO

Applicant Login Work Around

We are continuing to experience applicant login issues.

The following work around has been identified until this issue has been resolved. If you are successfully registered with Grants.gov and experiencing an issue logging in please follow these steps which will enable you to successfully log in.

1. Steps:Go to www.grants.gov
2. On the right side of the page, under "Applicants," click on "Applicant Login"
3. Click "Continue to Login"
4. Select "Forgot my Password"
5. Follow the instructions and then "enter your username"
6. You will then receive an email from grants.gov with a temporary password
7. Click on the link in the email and cut and paste the temporary password into the appropriate box
8. Complete your profile
9. After you complete your profile you will be asked to create your new password
10. You can use your old password if you like
11. Once you reset then you can successfully log in
    

If you have any questions please contact the Grants.gov Contact Center at 800-518-4726.

ALERT: Intemittent Applicant Login Issue

We have been getting intermittent reports of applicant login issues. Some applicants have taken 2 – 3 attempts before successfully getting into the system.

We are investigating the source of the login issue. We have discovered an LDAP query that is taking over 300 seconds to complete. This is consuming resources and slowing down logins. While we are investigating the cause of the improper query we have made available additional hardware resources to help improve the speed in the near term.

We do not have an estimated time to correct but will provide updates as we proceed.

Regards,
Grants.gov PMO

Grants.gov Build 2008 - 03

Grants.gov successfully deployed Build 2008 – 03 which increased the number of concurrent users logged into the system to 2,000. This increase will improve performance. The build changed Step 3 of the registration process where Grants.gov now maintains the username and password functionality internally versus utilizing a third party service provider.

The system was unavailable Saturday, February 7, through 12:01 AM Monday, February 9, 2009 at which point the system went live back into production.

There were no major issues with the deployment. There are two outstanding issues which we are currently working and will notify you once resolved. At this time we do not have an estimated time of resolution but we anticipate correcting shortly.

1. Applicant e-mail: This is an intermittent issue which has arisen during testing. When an applicant logs into the system and they go to manage profile, if they update their e-mail address the updated e-mail address is not taking effect. There is no message telling the applicant of this issue therefore they are unaware their e-mail address has not been updated. Any e-mails sent by the system will be sent to the applicant e-mail address that existed before the change.

2. Grantor Token: When Grantors register they receive an email message with an Activation Token. To associate another credential with this profile, Grantors need the Activation Token. Users who have been assigned the Manage Synopses role will receive a second email message with a user ID and password for use with the Grants.gov Email XML Synopsis service. This token is currently not working.

This morning we have received reports from our Contact Center that applicants are experiencing issues when they go to log in. We are currently working to identify the cause of this issue and will provide you with more details as they become available.

We appreciate your continued patience as we work towards resolving these open issues. If you have any questions please contact the Grants.gov PMO staff.

Regards,
Grants.gov PMO

Grants.gov will be unavailable from 12:00AM Saturday, February 7, 2009 through 8:00AM Monday, February 9, 2009.  Over this period Grants.gov will be implementing changes to improve overall system performance and update forms under System Build 2008-03.  Once the changes are implemented users should experience faster login times, improved system performance, clearer email notifications, and more choices when using the RSS feed.  When the system is brought back online additional information will be available at the Site Features and Enhancements page of Grants.gov.  Grants.gov will provide notice if there is any change to this implementation.        

UPDATE: Grantor Log in Issue Corrected

Working with the E-authentication office we have corrected the Grantor log in issue. Grantors can now successfully log into the system.

We apologize for any inconvenience this may have caused today. If you have any questions please contact the Grants.gov PMO staff.

Regards,
Grants.gov PMO

Grantor login problem

Grantor users are experiencing a login problem with all 3 CSPs. Registered grantor users with ORC and Employee Express IDs are being directed to the "Register for Grants.gov" page incorrectly. USDA users may experience the inability to login from the USDA site. we have contacted USDA E-Auth as well. All parties are working to resolve this problem as soon as possible.

ALERT: System Scheduled Down Time

Grants.gov is scheduled to deploy system Build 2008 – 03 this weekend. To deploy the build we need to take the Grants.gov system offline. The system will be unavailable Saturday, February 7, 2009 through Sunday, February 8, 2009. The system is scheduled to be brought back online at 11:59 PM EST February 8. If the deployment is ahead of schedule we will bring the system online sooner.

This build will increase the number of concurrent users up to 2,000 which will alleviate existing system slowness.

This build corrects the erroneous e-mail notifications issue. We need to conduct a test in production to formally validate this issue has been resolved.

Once this build is deployed the next priority is to correct the CCR data extract pull. We will provide you with more information as both these issues are resolved.

We appreciate your patience as we have worked towards deploying this build. If you have any questions please contact the Grants.gov PMO staff.

Grantors: USDA E-Auth is down

USDA E-Auth is experiencing a problem. USDA has posted a message on their site regarding the service problem they are experiencing. This service problem only impacts grantors attempting to access Grants.gov. ORC and Employee Express are operating.

Contact Center

If you exeperience extended wait times today due to the weather situation and you do not need immediate assistance please e-mail the Contact Center: support@grants.gov.

This will reduce some of the call volume.

ALERT: Contact Center Extended Call Wait Time

The Grants.gov Contact Centers physically reside in Kentucky and Texas. Yesterday, January 27, the Kentucky facility experienced a phone outage due to a Sprint fiber line outage which has since been corrected by Sprint. When the outage occurred all calls were forwarded to Texas. This resulted in increased wait times. A message was put on our IVR script indicating extended wait times for callers.

Today, January 28, the Texas facility has encountered a number of employees calling out for the day to due the impact of the ice storm there; schools and businesses are closed therefore Customer Support Representatives had to take off to take care of their families.

With the system slowness experienced over the last week calls have doubled. With the situation today as a result of the ice storm in Texas caller wait times will be extended. We have placed a message on the IVR script alerting customers. If you do not require immediate assistance please e-mail the Contact Center www.support@grants.gov

Regards,
Grants.gov PMO

ALERT: Contact Center Extended Call Wait Time

The Grants.gov Contact Centers physically reside in Kentucky and Texas. Yesterday, January 27, the Kentucky facility experienced a phone outage due to a Sprint fiber line outage which has since been corrected by Sprint. When the outage occurred all calls were forwarded to Texas. This resulted in increased wait times. A message was put on our IVR script indicating extended wait times for callers.

Today, January 28, the Texas facility has encountered a number of employees calling out for the day to due the impact of the ice storm there; schools and businesses are closed therefore Customer Support Representatives had to take off to take care of their families.

With the system slowness experienced over the last week calls have doubled. With the situation today as a result of the ice storm in Texas caller wait times will be extended. We have placed a message on the IVR script alerting customers. If you do not require immediate assistance please e-mail the Contact Center support@grants.gov


Regards,
Grants.gov PMO

ALERT: Grantor and Applicant login error resolved

The Error 404 experienced by users this morning has been resolved.

ALERT: Grantor and Applicant login error

The system is currently experiencing an "Error 404" for the grantor, applicant, and E-biz logins.

We are aware of the problem and working on the resolution. Once resolved, we will update this blog message.

Grants.gov System Status

At 8:50 a.m. this morning there were 850 users logged into the system. We anticipate a heavy load this week so we continue to recommend applicants submit off hours and early.

If an applicant calls our Contact Center and the system status is slow then the Customer Support Representative will give you the option to call back and to create a customer service ticket on your first call that you can reference when you call us back. When the system is slow there is a ripple effect to the Contact Center. The reports the Contact Center run take a long time when the system is slow.

We are scheduled to conduct round 3 of testing on Build 2008 - 03 this week and then determine a deployment date. This build increase the number of concurrent users to two thousand, hence alleviating the current system slowness users are experiencing.

Regards,
Grants.gov PMO

ALERT: Applicant and Grantor login problem resolved

The Error 404 message received by some users earlier this morning has been resolved. Site usage remains high and users may experience slow response or delays when logging into the system.

ALERT: Grantor and Applicant Login problem

The system is currently experiencing an "Error 404" for the grantor, applicant, and E-biz login.

Submission of applications is not affected by the issue. Applicants can successfully submit applications.

We are working to resolve this issue and will notify you once resolved.

ALERT: Applicant & Grantor Login Issue

The system is currently experiencing issues with both applicant and grantor login.

At this time the Apply functionality is working. Applicants are able to submit if they are able to log in.

We are working to resolve this issue and will notify you once resolved.

ALERT: System Slowness

Updated January 22, 2009

Grants.gov is experiencing extreme system slowness since we have 2,000 users logged into the system right now. If the grant opportunity you are applying for does not close today we recommend you submit off hours (after 6:00 p.m EDT) so those with closings today can successfully submit.

If you are unable to submit by your closing deadine please contact the Grants.gov Contact Center to create a customer service ticket identifying your trouble submitting.

Contact Center
800-518-4726

Grants.gov Daily E-mail Notifications of New Opportunities

Grants.gov was scheduled to terminate the daily e-mail notifications of Agency new grant opportunities on January 31, 2009. During the January 15, 2009 GEB the decision was made to keep the daily e-mail notifications so this service will not be terminated. We will continue the RSS feed so applicants will have two sources to view new opportunities.

If you have any questions please contact the Grants.gov PMO staff.

Regards,
Grants.gov PMO

ALERT - Grants.gov Applicant Login Issue

If you are an applicant who has renewed, modified or registered with the Central Contractor Registration (CCR) after December 31, 2008 and cannot login to Grants.gov please contact the Grants.gov Contact Center (1-800-518-4726/support@grants.gov).

Grants.gov is currently experiencing an issue pulling the nightly data extract from CCR. If you have verified your CCR registration is active and you call the Grants.gov Contact Center please have the following information available so that Grants.gov can manually enter your information into the Grants.gov system to activate your Grants.gov username and password.

DUNS, DUNS plus 4, legal business name, government business primary POC email address, M-PIN, registration date with CCR, applicant name, phone number and email address.

Grants.gov will create a customer service ticket for your call and notify you once your Grants.gov username and password are active.

Regards,
Grants.gov

ALERT: Testing Environment Unavailable

The Grants.gov testing environment (AT07) is currently unavailable and off-line. We are working on bring it back up and will alert you when you can begin testing.

Regards,
Grants.gov PMO