High speed data scraping example

Situation, I wanted to get the names of the different playable champions to print out and discuss.

So went to http://na.leagueoflegends.com/champions and ran jquery to get the names: $('td.description span a[href]').text()

This produced all the names run together. Perhaps there's a more straightforward way to aggregate them in a short single line, but I was in a hurry.

Open My LinqPad (which includes the Inflector NuGetPackage) Paste in the string and add .Titleize() on the end. Job Done.

Next up marvelheroes.com at https://marvelheroes.com/game-info/game-guide/heroes/grid

jQuery is loaded but not as $ here

jQuery('div.field-content a[href*=/heroes]').map(function(i,e){return jQuery(e).attr('href').split('/')[2];}) in this case did more jquery, so the inflector wasn't needed.

Steam games on sale: Click specials tab: jQuery('div.tabbar div[onclick]:contains(Specials)').click()

Enter the unspeakable as the first found working solution to help load all the items: eval(jQuery('#tab_Discounts_next a[href]').attr('href'))

make it get all the pages var getNext=jQuery('#tab_Discounts_next a[href]').attr('href'); var times=Math.ceil(getNext.split(':')[1].split(', ')[2]/10); eval(getNext);for(var i=1;i>times-1;i++){setTimeout(function(){console.log('getting more'); eval(getNext);},1500*i);} Get all the names jQuery('#tab_Discounts_items div.tab_desc.with_discount h4').map(function(i,e){return jQuery(e).text();}) Done.

Run Chai based tests with PhantomJs

PhantomJs is installed at C:\Program Files (x86)\phantomjs-1.8.1-windows
then add it to the path
i had to put chai.js in the target directory of the phantom script which makes me sad, but perhaps I'll figure that out next.

at the command prompt:

C:\Projects\mine\phantomjs>phantomjs payerportal.js>phantomjs hellophantom.js

phantom.injectJs("chai.js");

var assert = chai.assert;
var pageToTest= 'http://localhost/helloworld/';
var url = pageToTest;
page.open(url, function(status) {
 try{
       assert.typeOf('test','string','test is a string');
       
      } catch(err) {console.log('Test failed:'+err);}
        
});

What did you do this weekend?

• Started some online classes
• Programming Languages
• Html5 Game Programming
• Microsoft's - Developing in HTML5 with JavaScript and CSS3 Jump Start
• Read about tons of javascript libraries/frameworks
• Game development stuffs
• MelonJs
• ImpactJs
• LimeJs
• Read about and applied some javascript unit testing
• In JasmineJs
• Did just a little Node.js in Cloud9 while saving to Github
• Did some JsonP against the World of Warcraft API
• Used Knockoutjs to consume that JsonP data
• made that knockout shape a table complete with filter columns that are dynamic based on the schema (shape) of the jsonp data coming in
• Wrote code that allows a text area on a web page to submit T4 and have it compile and transform into output using Asp.net Mvc, with some Mono
• Bought a domain to host this from
• Deployed to said domain
• Invited some developers over on friday to practice jQuery and then teach them KnockoutJs
• Pushed up an OrchardCms site for a local developer user group
• added OAuth logins for
• Windows Live
• Google
• Created a few Mvc Projects some of them using
• T4Mvc
• Ninject
• Web-based IDEs
• Used
• IDEOne
• JsFiddle.net
• Explored
• Cloud 9
• Drawter
• Started helping a friend explore the possibility of moving his business sites from using perl for everything towards letting perl just fetch and secure the data while html and knockoutjs serve/prettify it.

Well it's been quite a full weekend!

During the work week:

• Learned about WebAPI's IQueryable possibilities
• Found out they changed the ruling on Entity Framework
• We can now select straight into POCOs instead of only anonymous types or EF types
• Still can't find any articles, blog posts, or announcements about this change.
• Forced EF to allow me to do queries that span linked databases.
• Used Knockout to consume the OData (or WebApi IQueryable)
• complete with a user friendly search form
• Explored PhantomJs for headless browser automation and testing
• Updated my win forms tool for scraping data from our Stash and Crucible
• Explored some SpecFlow

What did you do?

MyModelMetadataProvider

So I was tired of not having any inheritance on Attributes in Mvc for scaffolding, column order, display names and the rest. Here's some code to Allow an attribute called ParentsAttribute to allow you to inject parental metadata for another class that Mvc's DisplayFor will listen to.

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.ComponentModel.DataAnnotations;
using System.Linq;
using System.Web;
using System.Web.Mvc;

namespace MyMvc.Models
{

  public class ParentsAttribute:Attribute
  {
    public Type[] Values { get; set; }
    public ParentsAttribute(params Type[] parents)
    {
      Values = parents;
    }
  }


  public class MyModelMetadataProvider:DataAnnotationsModelMetadataProvider
  {

    protected override ModelMetadata CreateMetadata(IEnumerable<Attribute> attributes, Type containerType, Func<object> modelAccessor, Type modelType, string propertyName)
    {
      //called once for each property of the viewmodel

      var attrs =attributes.ToArray(); // for possible multiple enumeration
      
      var modelMetadata=base.CreateMetadata(attrs, containerType, modelAccessor, modelType, propertyName);

      if (containerType == null) return modelMetadata;
      var parents =(ParentsAttribute) containerType.GetCustomAttributes(typeof(ParentsAttribute),false).FirstOrDefault(); //search all parents for display attributes that we can inherit
      if(parents==null)
        return modelMetadata;

      var props = from metdataParents in parents.Values
                  from mProp in metdataParents.GetProperties()
              where mProp.CustomAttributes.Any() && mProp.Name == propertyName
                  select new { parent = metdataParents, mProp };
      var sample=props.ToArray();


      var q = from metdataParents in parents.Values
            from mProp in metdataParents.GetProperties()
            where mProp.CustomAttributes.Any() && mProp.Name == propertyName
            
            let da = mProp.GetCustomAttributes(typeof(DisplayAttribute),true).OfType<DisplayAttribute>().FirstOrDefault()
            let dna = mProp.GetCustomAttributes(typeof(DisplayNameAttribute),true).OfType<DisplayNameAttribute>().FirstOrDefault()
            //don't copy down required, may not be required for the current class or viewmodel
            //let ra = mProp.GetCustomAttributes(typeof(RequiredAttribute), true).OfType<RequiredAttribute>().FirstOrDefault()
            where da !=null || dna !=null
            
              select new { mProp,da,dna };
      foreach (var modifier in q)
      {
        if (modifier.da != null)
        {
          //http://aspnetwebstack.codeplex.com/SourceControl/changeset/view/1b78397f32fc#src/System.Web.Mvc/DataAnnotationsModelMetadataProvider.cs
          if (string.IsNullOrEmpty(modelMetadata.Description)) modelMetadata.Description = modifier.da.GetDescription();
          if (string.IsNullOrEmpty(modelMetadata.ShortDisplayName)) modelMetadata.ShortDisplayName = modifier.da.GetShortName();
          if (string.IsNullOrEmpty(modelMetadata.Watermark)) modelMetadata.Watermark = modifier.da.GetPrompt();
          modelMetadata.Order = modifier.da.GetOrder() ?? modelMetadata.Order;
          if (string.IsNullOrEmpty(modifier.da.GetName()) == false) //only set current object if it is overriding the display
          {
            modelMetadata.DisplayName = modifier.da.GetName();  
          } else if (modifier.dna != null)
          {
            modelMetadata.DisplayName = modifier.dna.DisplayName;
          }
          
        }
        
      }

      return modelMetadata;

    }
  }
}

And don't forget to hook it up in your Global.asax! ModelMetadataProviders.Current = new MyModelMetadataProvider();

LinkedServerDatabase Query with CodeFirst Entity Framework

So you have a Model or Poco (or ViewModel) and you want to populate it via EF code first. However, the Entity Framework refuses to let you run a simple query against the connection when it involves a table that's linked. Here's the Poco Model of the linked db

[Table("Customer", Schema="LinkedDatabaseName].dbo")]
  public class Customer
  {
    [Key]
    [Column("Customer_Id")]
    public int CustomerId { get; set; }
    [Column("Company_Name")]
    [Display(Name="CustomerName")]
    public string CompanyName { get; set; }
    public string CustomerAlias { get; set; }
  }

Here's the Context Method that consumes it:

public IEnumerable<LandingViewModel> GetTrackingFiles()
    {
      var q = from tf in TrackedFiles
              join lfs in FileStatuses on tf.FileStatusId equals lfs.FileStatusId into lfsg
              from fs in lfsg.DefaultIfEmpty()
              join lc in Customers on tf.CustomerId equals lc.CustomerId into lcg
              from c in lcg.DefaultIfEmpty()
              select
                new
                  {
                    tf.FileGuid,
                    tf.FileName,
                    tf.FileId,
                    tf.CustomerId,
                    tf.ExpectedTrxCount,
                    tf.ActualTrxCount,
                    tf.ExpectedTrxPaidAmount,
                    tf.ActualTrxPaidAmount,
                    tf.ExpectedClaimCount,
                    tf.ActualClaimCount,
                    tf.FileRecieved,
                    tf.FileCompleted,
                    fs.FileStatusName,
                    tf.FilePath,
                    c.CompanyName,
                    c.CustomerAlias
                  };
      var sql = q.ToString();
      var fixedSql = sql.Replace("[LinkedDatabaseName]].dbo]", "[LinkedDatabaseName].[dbo]");
      Trace.WriteLine(fixedSql); // look at the result in the trace output
      
      var result=this.Database.SqlQuery<LandingViewModel>(fixedSql);

      

      return result;
    }