Indexing external data with solr

This will provides a glimpse to index external data with solr. Here external data sources files like CSV, json and xml.

Prerequisites:
Solr needs to be installed in your system or server.

Creation of Core in Solr

A Solr core is basically an index of the text and fields found in documents. Core needs to be created to index data in solr and need to change the schema file with respect to the external data source.

Steps to create Core

Open "Command Prompt" and navigate to path solr installation bin folder.In my case, it is “D:\RapidSitecore\Solr\solr-6.6.1\bin” and type the below command to create core in solr.

Solr create –c xmldata

Here “xmldata” is the name of the core.

Check below snapshot for the successful creation of core.

Once after the successful creation of the core, you can find it on the solr admin portal.
We can also create core through UI. Goto solr admin --> core admin --> click on add core and provide the name of the core and then click on Add core. It will create a core in solr.

How to index XML data on Solr

Let us take one xml file with name manufacturers.xml and this data to be index with solr and it needs to be present in the following path. The sample data of xml file is as below.

D:\RapidSitecore\Solr\solr-6.6.1\example\exampledocs

<add>

<doc>

<field name="id">adata</field>

<field name="compName_s">A-Data Technology</field>

<field name="address_s">46221 Landing Parkway Fremont, CA 94538</field>

</doc>

<doc>

<field name="id">apple</field>

<field name="compName_s">Apple</field>

<field name="address_s">1 Infinite Way, Cupertino CA</field>

</doc>

</add>

Now we will make changes to the core schema file with respect to the XML data.
Open “managed-schema” file under D:\RapidSitecore\Solr\solr-6.6.1\server\solr\xmldata\conf and add below fields under < uniqueKey>id< /uniqueKey> in the schema file

Now go to command prompt and type below command to index data in solr.

java -Dc=xmldata -Dtype=application/xml -jar post.jar manufacturers.xml

Here xmldata is the name of the core

Manufactures.xml is the name of the xml file.

Once if index got created successfully then you can see as below in the command prompt

Now navigate to Solr admin and goto query and click on execute query and you will see the data solr as per below.

How to index CSV data on Solr

Create a core as mentioned in the section Creation of Core in Solr Let us take one csv file with name books.csv and this data to be index with solr and it needs to be present in the following path. Sample data of csv file is as below.

D:\RapidSitecore\Solr\solr-6.6.1\example\exampledocs

id,cat,name,price,inStock,author,series_t,sequence_i,genre_s

0553573403,book,A Game of Thrones,7.99,true,George R.R. Martin,"A Song of Ice and Fire",1,fantasy

0553579908,book,A Clash of Kings,7.99,true,George R.R. Martin,"A Song of Ice and Fire",2,fantasy

Now we have make changes to the core schema file with respect to the csv data.
Open “managed-schema” file under D:\RapidSitecore\Solr\solr-6.6.1\server\solr\csvdata\conf and add below fields under < uniqueKey>id< /uniqueKey> in the schema file

Now go to command prompt and type below command to index data in solr.

java -Dc=csvdata -Dtype=application/csv -jar post.jar books.csv

Here csvdata is the name of the core

books.csv is the name of the csv file.

Once if index got created successfully then you can see as below in the command prompt

Now navigate to Solr admin and goto query and click on execute query and you will see the data solr as per below.

How to index json data on Solr

Create a core as mentioned in the section Creation of Core in Solr
Let us take one json file with name books.json and this data to be index with solr and it needs to be present in the following path. Sample data of json file is as below.

D:\RapidSitecore\Solr\solr-6.6.1\example\exampledocs

[

{

"id" : "978-0641723445",

"cat" : ["book","hardcover"],

"name" : "The Lightning Thief",

"author" : "Rick Riordan",

"series_t" : "Percy Jackson and the Olympians",

"sequence_i" : 1,

"genre_s" : "fantasy",

"inStock" : true,

"price" : 12.50,

"pages_i" : 384

}

{

"id" : "978-1423103349",

"cat" : ["book","paperback"],

"name" : "The Sea of Monsters",

"author" : "Rick Riordan",

"series_t" : "Percy Jackson and the Olympians",

"sequence_i" : 2,

"genre_s" : "fantasy",

"inStock" : true,

"price" : 6.49,

"pages_i" : 304

}

]

Now we have will make changes to the core schema file with respect to the json data.
Open “managed-schema” file under D:\RapidSitecore\Solr\solr-6.6.1\server\solr\jsondata\conf and add below fields under < uniqueKey>id< /uniqueKey> in the schema file

Now go to command prompt and type below command to index data in solr.

java -Dc=jsondata -Dtype=application/json -jar post.jar books.json

Here jsondata is the name of the core

books.json is the name of the json file.

Once if index got created successfully then you can see as below in the command prompt

Now navigate to Solr admin and goto query and click on execute query and you will see the data solr as per below.

Troubleshooting

Verify data is valid and in the correct format.
If you feel everything is correct and still getting error in the creation of core or creating indexes then restart solr services in service.msc.

That's all for now, keep learning 😆

Techie Ticker

Search This Blog

Step-by-Step Publishing Service Configuration

Indexing external data with solr

Comments

Post a Comment