Namaste 🙏 ,
Welcome to the eleventh weekly blog of GSoC’20 for CDLI.
This week, I completed the SQL query for an advanced search index. After indexing the data, the main task was to handle multiple fields of Advanced Search forms and integrating it with ES query to fetch results.
After integrating form and forming query in the required format, an issue arose regarding maximum hits (=10000) fetched from ES in one go. This issue was not addressed in Simple Search and postponed to address in the advanced search. So the rescuer was Scroll API, to fetch all the hits (> 10000) on the cost of heap memory. After optimising ES query, advanced search query was finally in a place with the optimised approach.
In the upcoming week, pagination for a large set of results will be implemented along with a filter option for search results.
At a time from ElasticSearch, 10000 results can be fetched and to fetch more results comes at the cost of heap memory. The current index for advanced search contains almost 20 fields which require more memory for ElasticSearch to execute the search query. Finally, it was optimized by fetching only required fields in search results.
| # | Day | Date | A short description of the work done |
|---|---|---|---|
| 1 | Monday | 2020/08/10 | a. WIP: Advance Search (SQL Query for ES index). |
| 2 | Tuesday | 2020/08/11 | a. WIP: Advance Search (SQL Query for ES index). |
| 3 | Wednesday | 2020/08/12 | a. WIP: Advance Search (SQL Query for ES index). |
| 4 | Thursday | 2020/08/13 | a. WIP: Advance Search (Backend). |
| 5 | Friday | 2020/08/14 | a. WIP: Advance Search (ElasticSearch Query). |
| 6 | Saturday | 2020/08/15 | a. WIP: Advance Search (ElasticSearch Query). |
| 7 | Sunday | 2020/08/16 | a. WIP: Advance Search. |