Learning Solr Code: Import Data to Solr

The main logic is at solr.servlet.SolrRequestParsers.parse(SolrCore, String, HttpServletRequest)

public SolrQueryRequest parse( SolrCore core, String path, HttpServletRequest req ) throws Exception
  {
    SolrRequestParser parser = standard;
  
    ArrayList streams = new ArrayList(1);
    SolrParams params = parser.parseParamsAndFillStreams( req, streams );
    SolrQueryRequest sreq = buildRequestFrom( core, params, streams );
    sreq.getContext().put( "path", path );
  }  

There are multiple SolrRequestParser implementations: FormDataRequestParser, MultipartRequestParser, RawRequestParser, SimpleRequestParser, StandardRequestParser. By default, it uses StandardRequestParser.

In StandardRequestParser.parseParamsAndFillStreams, if it is GET or HEAD request, it will parse the query string, and create a SolrParams: please refer about how it parses the query string.

If it is a POST request, for normal post request, StandardRequestParser.parseParamsAndFillStreams will use FormDataRequestParser to parse the form data, and create a SolrParams.
The following curl request will be handled by FormDataRequestParser.
curl -d "stream.body=<add><doc><field name='contentid'>content1</field></doc></add>&clientId=client123&batchId=1" http://host:port/solr/update

If the data is uploaded as a file, like below:
curl http://host:port/solr/update -F "fieldName=@data.xml"
The fieldName doesn't matter and can be anything.

StandardRequestParser.parseParamsAndFillStreams will use MultipartRequestParser, which will use apache commons to create fileupload.FileItem, then create a servlet.FileItemContentStream.
How it determines whether the request is multipart?
ServletFileUpload.isMultipartContent(req), whether the contentType starts with "multipart/".

For a POST request, if the request is not format mentioned before, it will use RawRequestParser which creates a servlet.HttpRequestContentStream from the request.

Then in SolrRequestParsers.buildRequestFrom, it will get stream.file, stream.body, stream.url, and constructs ContentStreamBase.FileStream/StringStream/URLStream. The file stream.file points to must be a local file to Solr server.
Subclasses of ContentStreamBase
HttpRequestContentStream
Wrap an HttpServletRequest as a ContentStream
public InputStream getStream() throws IOException {
return req.getInputStream();
}
FileItemContentStream
Wrap a org.apache.commons.fileupload.FileItem as a ContentStream
ContentStreamBase.FileStream
ContentStreamBase.URLStream
ContentStreamBase.StringStream
DocumentAnalysisRequestHandlerTest.ByteStream
Using curl to send request to Solr
curl -d "stream.body=<add><doc><field name=\"id\">id1</field></doc></add>&clientId=client123" http://host:port/solr/update
curl -d "stream.body=<add><commit/></add>&clientId=client123" http://host:port/solr/update
Error:
In this case, have to add "" for the value of -d, as the value contains special characters, like <, otherwise it will report error:
curl -d stream.body=<add><doc><field name=\"id\">id1</field></doc></add>&clientId=client123 http://host:port/solr/update
< was unexpected at this time.

For the stream body, have to use " to enclose property name, like \"id\". The following request will fail:
curl -d "stream.body=<add><doc><field name=id>id1</field></doc></add>&clientId=client123" http://host:port/solr/update
org.apache.solr.common.SolrException: Unexpected character 'i' (code 105) in start tag Expected a quote
at [row,col {unknown-source}]: [1,23]
Caused by: com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected character 'i' (code 105) in start tag Expected a quote
at [row,col {unknown-source}]: [1,23]
at com.ctc.wstx.sr.StreamScanner.throwUnexpectedChar(StreamScanner.java:648)

Correct Usage:
Use "" to enclose the value of -d.
Use \ to escape specail characrts, " to \", \ to \\. 
curl -d "stream.body=2,0,1,0,1,\"c:\\\",1,0,\"c:\",0,1,16 %0D%0A 2,0,1,0,1,\"x:\\\",2,0,\"x:\",0,1,16 &separator=,&fieldnames=omiited&literal.id=9000&stream.contentType=text/csv;charset=utf-8&commit=true" http://localhost:8080/solr/update/csv
Code:

private final boolean com.ctc.wstx.sr.BasicStreamReader.handleNsAttrs(char)(char c)
 // And then a quote:
 if (c != '"' && c != '\'') {
  throwUnexpectedChar(c, SUFFIX_IN_ELEMENT+" Expected a quote");
 }

Upload csv content
curl -d "stream.body=id1&clientId=client123&fieldnames=id" http://host:port/solr/update/csv
Upload XML File
curl -F "fieldName=@data.xml" http://host:port/solr/update
curl -F "fieldName=@data.xml;type=application/xml" http://host:port/solr/update
;type=application/xml set MIME content-type of the file.
curl -F fieldName=@data.xml -F clientId=client123 -F &batchId=2 http://host:port/solr/update
Have to use multiple -F for multiple form data, format -F "key1=value1&key2=value2" doesn't work - this will only set one pair, value of key1 is value1&key2=value2.
Delete Data
curl -d "stream.body=<delete><query>*:*</query></delete>&commit=true" http://host:post/solr/update
Curl Usage
-d, --data <data>
(HTTP) Sends the specified data in a POST request to the HTTP server.
--data-binary <data>
(HTTP) This posts data exactly as specified with no extra processing whatsoever.
curl http://host:port/solr/update -H "Content-Type: text/xml" -d @C:\jeffery\data.xml
-F, --form <name=content>
curl -F password=@/etc/passwd www.mypasswords.com
curl -F "name=daniel;type=text/foo" url.com

Set the Request Method: -X POST
Set Request Headers: -H "Authorization: OAuth 2c4419d1aabeec"
View Response Headers: -i
Debug request: -v

-o (lowercase o) the result will be saved in the filename provided in the command line
-O (uppercase O) the filename in the URL will be taken and it will be used as the filename to store the result
Follow HTTP Location Headers with -L option

To POST to a page
curl -d "item=bottle&category=consumer&submit=ok" www.example.com/process.php

Referer & User Agent
curl -e http://some_referring_site.com http://www.example.com/
curl -A "Mozilla/5.0 (compatible; MSIE 7.01; Windows NT 5.0)" http://www.example.com

Limit the Rate of Data Transfer
curl --limit-rate 1000B -O http://www.gnu.org/software/gettext/manual/gettext.html

Continue/Resume a Previous Download: -C -
curl -C - -O http://www.gnu.org/software/gettext/manual/gettext.html

Pass HTTP Authentication in cURL
curl -u username:password URL

Download Files from FTP server
curl -u ftpuser:ftppass -O ftp://ftp_server/public_html/xss.php

List/Download using Ranges
curl ftp://ftp.uk.debian.org/debian/pool/main/[a-z]/

Upload Files to FTP Server
curl -u ftpuser:ftppass -T myfile.txt ftp://ftp.testserver.com
curl -u ftpuser:ftppass -T "{file1,file2}" ftp://ftp.testserver.com
curl ftp://username:password@example.com

Use Proxy to Download a File
curl -x proxysever.test.com:3128 http://google.co.in

References
http://curl.haxx.se/docs/manpage.html
http://curl.haxx.se/docs/httpscripting.html
9 uses for cURL worth knowing
6 essential cURL commands for daily use
9 uses for cURL worth knowing
15 Practical Linux cURL Command Examples (cURL Download Examples)