The question:
I’m working on an application that needs to get the source of a web page from a link, and then parse the html from that page.
Could you give me some examples, or starting points where to look to start writing such an app?
The Solutions:
Below are the methods you can try. The first solution is probably the best. Try others if the first one doesn’t work. Senior developers aren’t just copying/pasting – they read the methods carefully & apply them wisely to each case.
Method 1
You can use HttpClient to perform an HTTP GET and retrieve the HTML response, something like this:
HttpClient client = new DefaultHttpClient();
HttpGet request = new HttpGet(url);
HttpResponse response = client.execute(request);
String html = "";
InputStream in = response.getEntity().getContent();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder str = new StringBuilder();
String line = null;
while((line = reader.readLine()) != null)
{
str.append(line);
}
in.close();
html = str.toString();
Method 2
I would suggest jsoup.
According to their website:
Fetch the Wikipedia homepage, parse it to a DOM, and select the headlines from the In the news section into a list of Elements (online sample):
Document doc = Jsoup.connect("http://en.wikipedia.org/").get();
Elements newsHeadlines = doc.select("#mp-itn b a");
Getting started:
Method 3
This question is a bit old, but I figured I should post my answer now that DefaultHttpClient, HttpGet, etc. are deprecated. This function should get and return HTML, given a URL.
public static String getHtml(String url) throws IOException {
// Build and set timeout values for the request.
URLConnection connection = (new URL(url)).openConnection();
connection.setConnectTimeout(5000);
connection.setReadTimeout(5000);
connection.connect();
// Read and store the result line by line then return the entire string.
InputStream in = connection.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(in));
StringBuilder html = new StringBuilder();
for (String line; (line = reader.readLine()) != null; ) {
html.append(line);
}
in.close();
return html.toString();
}
Method 4
public class RetrieveSiteData extends AsyncTask<String, Void, String> {
@Override
protected String doInBackground(String... urls) {
StringBuilder builder = new StringBuilder(100000);
for (String url : urls) {
DefaultHttpClient client = new DefaultHttpClient();
HttpGet httpGet = new HttpGet(url);
try {
HttpResponse execute = client.execute(httpGet);
InputStream content = execute.getEntity().getContent();
BufferedReader buffer = new BufferedReader(new InputStreamReader(content));
String s = "";
while ((s = buffer.readLine()) != null) {
builder.append(s);
}
} catch (Exception e) {
e.printStackTrace();
}
}
return builder.toString();
}
@Override
protected void onPostExecute(String result) {
}
}
Method 5
Call it like
new RetrieveFeedTask(new OnTaskFinished()
{
@Override
public void onFeedRetrieved(String feeds)
{
//do whatever you want to do with the feeds
}
}).execute("http://enterurlhere.com");
RetrieveFeedTask.class
class RetrieveFeedTask extends AsyncTask<String, Void, String>
{
String HTML_response= "";
OnTaskFinished onOurTaskFinished;
public RetrieveFeedTask(OnTaskFinished onTaskFinished)
{
onOurTaskFinished = onTaskFinished;
}
@Override
protected void onPreExecute()
{
super.onPreExecute();
}
@Override
protected String doInBackground(String... urls)
{
try
{
URL url = new URL(urls[0]); // enter your url here which to download
URLConnection conn = url.openConnection();
// open the stream and put it into BufferedReader
BufferedReader br = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
while ((inputLine = br.readLine()) != null)
{
// System.out.println(inputLine);
HTML_response += inputLine;
}
br.close();
System.out.println("Done");
}
catch (MalformedURLException e)
{
e.printStackTrace();
}
catch (IOException e)
{
e.printStackTrace();
}
return HTML_response;
}
@Override
protected void onPostExecute(String feed)
{
onOurTaskFinished.onFeedRetrieved(feed);
}
}
OnTaskFinished.java
public interface OnTaskFinished
{
public void onFeedRetrieved(String feeds);
}
Method 6
If you have a look here or here, you will see that you can’t do that directly with android API, you need an external librairy…
You can choose between the 2 here’s hereabove if you need an external librairy.
Method 7
One of the other SO post answer helped me. This doesn’t read line by line; supposingly the html file had a line null in between. As preRequisite add this dependancy from project settings “com.koushikdutta.ion:ion:2.2.1” implement this code in AsyncTASK. If you want the returned -something- to be in UI thread, pass it to a mutual interface.
Ion.with(getApplicationContext()). load("https://google.com/hashbrowns") .asString() .setCallback(new FutureCallback<String>() { @Override public void onCompleted(Exception e, String result) { //int s = result.lastIndexOf("user_id")+9; // String st = result.substring(s,s+5); // Log.e("USERID",st); //something } });
Method 8
public class DownloadTask extends AsyncTask<String, Void, String> {
@Override
protected String doInBackground(String... urls) {
String result = "";
URL url;
HttpsURLConnection urlConnection = null;
try {
url = new URL(urls[0]);
urlConnection = (HttpsURLConnection) url.openConnection();
BufferedReader br = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
String inputLine;
while ((inputLine = br.readLine()) != null)
{
// System.out.println(inputLine);
result += inputLine;
}
br.close();
return result;
} catch (Exception e) {
e.printStackTrace();
return "failed";
}
}
}
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
DownloadTask task = new DownloadTask();
String result = null;
try {
result = task.execute("https://www.example.com").get();
}catch (Exception e){
e.printStackTrace();
}
Log.i("Result", result);
}
All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0