About two weeks ago, NITI Aayog launched the National Data and Analytics Platform (NDAP) for open public use. He said this platform will host foundational datasets from various government agencies and provide tools for analysis and visualization. A beta was released last year in August, which gave access to a limited number of users for testing and feedback. The NDAP seeks to “democratize” data delivery by solving many of the challenges that currently exist in the open data universe.
Object Technology Solutions India (OTSI) developed NDAP. In an exclusive interaction with Analytics India Magazine, Chandra Talluri, CEO of OTSI, talks more about the technology that powers NDAP.
OTSI offers its expertise in four main areas of technology practice:
- Digital transformation
- Data and analytics – data insights, visualization
- Automation – test automation and RPA
- Disruptive technology (more emphasis on R&D)
Unique technology partner for NDAP
NITI Aayog had issued a tender for hiring the consulting firm that can provide technology solution architecture for this project. “It’s a more results-oriented project. OTSI worked with the NITI Aayog team and understood various aspects of data in the public sector and open data sources. We proposed the complete solution, from the design to the architecture of the solution, the approach to take and how to approach the milestones. Overall, we are the only technology partner involved in this project,” Talluri said.
Technology Stack – AWS, Python and more
Although the entire technology stack cannot be revealed at this time, Talluri informs that the overall solution is hosted on AWS. OTSI used many components in scraping-related sourcing, mostly done on Python-specific algorithms. On the database side, it used multiple databases. There have also been vector database applications. The AWS RedShift database was also used for specific purposes. On the front-end, it is a ReactJS-based application.
Big names participated, OTSI took the advantage thanks to past records
“To be honest, it’s one of the most prestigious projects. Many big names participated in the tender. It was primarily aimed at Tier 1 partners. OTSI also has a good track record of delivering analytics projects across various public sectors and government departments. We have experience working with the public sector and government departments, our understanding of working with public datasets, adds Talluri.
Ministry of Corporate Affairs, Ministry of Home Affairs are former clients
OTSI worked with the Ministry of Corporate Affairs (MCA) to set up a business intelligence/data mining and analysis unit within the ministry and generate custom statistical reports using business intelligence tools. intelligence, data mining and analytics to be used by government departments for policy making. and for regulatory purposes. The main priority of this project is to demonstrate the potential of data mining, visualization and interactive reporting on company filing forms submitted by companies to the MCA21 system. OTSI has also worked with the Home Office and PM Division (MHA) for ICJS – Interoperable Criminal Justice System. It has also established partnerships with state governments.
Similar platforms have limited data sets
Similar platforms have existed before, so what’s so special about NDAP? AIM asked Talluri.
Talluri informs that in many other portals very limited datasets are available. They are also not updated regularly. “They took some of the open data sources and brought them into the platform. There is no mechanism to take all this data and get periodic updates in terms of supply. The number of data sources we have covered on NDAP is over 30,000 (more to come) by scraping these websites. The source is quite broad here,” he says.
Image: Home – NDAP (niti.gov.in)
Don’t think that the “merger” of two different datasets is available on other similar government platforms
In NDAP, there is a feature called “merging” datasets. One of the main challenges OTSI had to overcome was the algorithms that OTSI built to make the merger possible, Talluri informs.
Talluri gives an example:
Suppose you want to retrieve education data sets and census data and merge the two. India is going through many changes including geographical changes with the formation of new states. Telangana was not there before 2014. How to merge these datasets when you want to study Telangana districts and education KPIs before 2014?
Talluri says, “We have developed methods, including time series and geography-based algorithms, and we are able to split this data at district level based on geographic region to also give information before 2014. We also plan to bring it down to the village level.
More datasets will be added
It is an ongoing process. Talluri informs: “Currently, we have 47 ministries and 14 sectors have been covered. We will be adding more datasets from each ministry. It’s also about getting access. Right now we are exploring and getting the data. Some departments have exposed APIs while others have not. Some ministries have published the data in PDF; we also read these files. Some are stand-alone PDF documents. Some are published in the dashboards, and we also explore from the dashboards. »
Is NITI Aayog behind in the game?
The United States, United Kingdom and many other Western countries have these types of portals which the government has taken and tried in various ways. The Indian government, too, has tried similar things in the past, but may not have had the expected results. Now, NITI has taken on the responsibility of providing a more holistic view through NDAP.
This portal is not only aimed at ordinary citizens, but will also greatly help policy makers. Research institutes such as IITs and IIMs will greatly benefit from their assistance. Instead of researchers scrambling to find the data, it will be available to them, at their disposal. The NDAP aims to give ease of use to users who are probably not as familiar with the technology.
Learn how the NDAP Portal uses the merge feature.