Tuesday, February 25, 2014

My quest for commercial free recorded TV...

I've been on this quest for quite some time and have recently sank additional time into its partial successful.  I say partial because it would appear that nothing in the world of network television is absolute.  By that I mean commercials.

This is more of a how to and extension onto my PVR write up.  This takes into effect that you have a running Win7 or Win8 Media Center (MC) that is recording your TV shows.  To reiterate I despise having to rent set-top boxes from my carrier.  They are incredibly costly to rent over time.  With the FCC changes to CableCards (MCard) this provides a reasonable rate to acquire HDTV.  Through the use of PVR's in connection with your rented MCard you can have your TV the way you want it, cheaper.  There might be more than one way to get your recorded TV.  You could pay for your own DVR's such as Moxi, build your own or if you already have a PC add a TV Tuner such as Ceton PCIe or USB Tuners.

I chose the PC using a Ceton PCIe Tuner with Win8 MC.  Reasoning for the choice was the existing XBOX360 throughout the house act as MC Extenders.  Notice I wrote XBOX360 (360) and not XBOXOne (XB1).  XB1 does not come with extender capabilities or DLNA.  Thats right you heard me correctly.  So get to your local pawn shop or used game store to pick up a 360.  There has been no official word I know of to date that Microsoft is enabling the feature on XB1.  Why you ask?  MC is apparently a dieing feature.  Win7 came with MC automatically.  Win8 you have to buy it extra.  Its entirely possible that the next OS Microsoft puts out MC might not be available at all.  It appears MC users only account for 5% of the population therefore not profitable.  So what should you do?  Well Win8's lifecycle is quite a ways off at 10 Jan 2023 so in the next nine years I should be able to save approximately $6480 in equipment rentals and that's only putting that at $60 per month.

So in my quest to have commercial free TV...

I've looked at MC-TVConverter and MCEBuddy.  The reason I went with MCEBuddy was because of support.  It would appear MC-TVConverter was last updated in 2011 as of version 2.1.0.  As of this writing that was three years ago.  I have no idea if the project is still ongoing.

I needed a program that would run as a service on my PC and convert my TV shows to a usable format.  The .wtv file format is very large in size and is not easily taken to mobile devices or streamed.  One thing also that needs to be discussed when dealing with recordings is timing.  I think by default MC set the default recording at 2 minute prior and 3 minutes after.  I have set my recorder to start on time and off on time.  Recording prior and after has a way of screwing with the commercials.  Also make sure your computer clock on the Media Center is set to update routinely to keep the right time and not to magically go offset.


By usable format/container I mean .mp4, .m4v or even .avi.  I'm on the road occasionally and it would be nice to copy the TV show to my mobile device and watch it while say on the plane, exercising on the treadmill, or what ever.

MCEBuddy provides that.  MCEBuddy is free but there is also a latest and greatest version that if you donate, to date $20, you will get the latest version.  Now I decided to write this while I was testing MCEBuddy so I'm using the free version typically one version older than the latest.  I don't need to explain MCEBuddy's feature because you can get that from their page.

I installed MCEBuddy on my Win8 MC.  The default install settings.  Nothing special there.  Now I will advise if you are upgrading, word to the wise uninstall the old version before installing the new.

When using MCEBoddy there are a number of components that work with MCEBuddy.  Namely Comskip since we're dealing with the focus of this rambling.  Comskip is a free program and is already housed within MCEBuddy.  MCEBuddy gives you two choices for commercial skipping.  ShowAnalyzer and Comskip.  At the time of this writing to my knowledge ShowAnalyzer does not have any active support to continue the project.  Comskip however is actively supported.  You have the ability to purchase a donated version for $10 or more which gives you access to a version that supports .wtv and will also process it faster.  For this purpose if you want to scan your .wtv files and determine where the commercials are within your video and trouble shot you will have to purchase the donator version of Comskip.  I donated and received my login and password promptly.  The author does state that this is a manual process and could take time to receive.

The research that I have done has been merely web related of what I could figure out.  Bare in mind I am not an expert in this field by any stretch of the mind so proceed at your own risk.  Now Comskip has a number of helpful pages available to describe the different options available.  Setting up... I decide to create a directory on my MC and copied a test .wtv file.


I felt the TMZ show was pretty difficult to narrow down the commercials and remove.  Reason being there is a lot of fast pace video.  I have also extracted my donator Comskip version.


The file of importance here is comskip.ini and the comskipgui.exe.  Notice I have made a copy of the comskip.ini and renamed it comskip.ini.orig.  This is merely to protect the original configuration file.  Additionally its nice to create a shortcut of the comskipgui.exe on the desktop.  This will easy our ability to drag and drop video file etc on to the program for faster processing.  You can download my comskip.ini from here with the mods already completed for TMZ.  Overwrite the existing comskip.ini.



By default the default comskip.ini does not produce the .log file that we need.  Modify or add the following lines within the comskip.ini.  Mine they already exist. The file is read linear so each line is read.  The ";" only comments out after input so dont start the line with read format ie. "; output_mls=0 ; blah blah" because it will be read first in and have the second line "output_mls=1 ; blah blah" because the =0 version will be the one assumed.  No spaces between the call = number.

verbose=10 (This should be 10 for available logging.  When complete we should make this 0 when in production mode.)

delete_logo_file=0 (This does not remove the .txt file of the logo rendition.)

output_framearray=1 (This is believe is the .csv file that is created.)

output_debugwindow=1 (This I believe is the GUI window to see the video.)

This is already done on my comskip.ini.


Now switch over to your test directory and drag and drop the .wtv video file onto your shortcut comskipgui.exe.  A window will appear processing the video to detect commercials.  Mind you this is the default comskip.ini with added logging or mine with mods depending on which one you used.



The color graph lines mean tested aspects of the video like audio, logo, and black screens.  Once the video has completed we can take a look at what was detected.

Default comskip.ini:


Notice the bar across the top of the window.  These are the recognized commercials.  This is however not completely correct.



Now I am focused on the second commercial area just to the right.  Notice the pink line, not horizontal but vertical.  This is my location in the video.  Now the screen shows clearly a commercial but the black and grey bar do not extend beyond indicating a commercial cut area.


Lets try to fix this by modifying some of the comskip.ini settings.  Close the graphic window above.  There will also be a command prompt windows behind and that needs to be closed as well.  Now I have done a little research into individual comskip.ini files that others have created here.

The first line is probably one of the most important.  I recommend only changing one thing at a time and than process the video again or at least the .csv file for faster output. I'll explain but lets modify the .ini first. and explain the numbering.

1=black frame, 2=logo, 4=scene change, 8=fuzzy logic, 16=closed captions, 32=aspect ration, 64=silence, 128=cutscenes, 255=all

The default is 43.  Clearly 43 is not listed.  This is merely an addition issue.  The closest number to 43 is 32 + 8 + 2 + 1 = 43.  So this means commercial detection is using aspect ration, fuzzy logic, logo, and black frame to detect commercials.

Now again doing some research on other peoples work here the FOX.ini detect_method=119.  That means:

64=silence
32=aspect ration
16=closed captions
4=secen change
2=logo
1=black frame
119 total

TMZ is a FOX program.

The only other line I modified was the min_show_segment_size=4 to 90.  Sometimes some commercials aren't recognized and that insists that when a commercial starts it will be at least 90 seconds long but could be longer.  Reason being was the point above and it did not detect a portion of the remaining commercials.

Now if you've used my comskip.ini just replace yours.  You still have a copy .orig.

Notice the files in your test area.  The important one now is the .csv file.  This file when added to comskipgui.exe shortcut will reprocess without having to go through all the video.  You only have to re-run the video if you change anything dealing with the logo or brightness.


Drag and drop the .csv file onto the comskipgui.exe.  It will process the video with the new conskip.ini.  Notice now the commericial area was originally like this:


Looks like this now at detect_method=119:


Looks like this now at detect_method=117 (without logo detection -2).  Note that logo detection is important in network shows.  Now for removing commercial in shows like Black Sails, STARZ.  Yes no commericials but you still want to remove the highlights before and after.  This would not require logo detection because there aren't any.


The logo in the case of TMZ is important.

Now this isn't fool proof but adjustments like this can make your commercial automated efforts better.  What I have notice is at the end of TMZ during the final credits the logo does not appear and confuses Comskip into thinking its still a commercial.  Notice the red box looking for the logo. Control+g removed the graphic lines bottom of the linked page.  F1 provides a legend.


I've found it very difficult to attempt and manipulate the settings to fix this ie. brightness, black screen, etc.  Ultimately I decide to keep the last 360 seconds of the show.  Remember I'm recording on time to on time.  Not any time extra before or after the show.


Now I've decide to do these config files for each show.  Others have done it via channel.  Reasons being that if for any reason these need to change I know that if the config file works with multiple then changing it wont effect the other shows.  So I decided one for one.  This will also minimize efforts under MCEBuddy as well later on.

Now that I think I have the comskip.ini file narrow down to a possible match what can I do with it now?

Time to add to MCEBuddy.  After installing MCEBuddy open the application.


The application will appear running like so.


To process our .wtv TMZ file or what ever show to decided to edit the comskip.ini file for.

Select --> Settings


Now you can see monitor location and conversion tasks already outline in my MCEBuddy.  Yours will be blank.

First lets set the monitor location.

Select --> Add


Monitor name --> Recorded TV

Monitor path --> [This is the location were your Recorded TV or .wtv files are located]

Search pattern --> [This is empty. You could put *.wtv but empty will search all]

OK

Now add a conversion task.



Task name --> TMZ [or whatever you comskip.ini file is for]

Profile --> MP4 HQ seems to work with all of my devices (360, Android, PS3, O!play, Sharp TV DLNA, etc)

Destination --> [decide where your output video file should go...I choose a separate hard disk drive (HDD) from the Recorded TV because it will read from the Recorded TV HDD and write to a separate HDD]

Remove ads --> Yes (Comskip)

OK

The TMZ task should be present.


Now I decided to store my comskip.ini files separate from MCEBuddy installation files as MCEBuddy is routinely upgraded.  So decide on a location and save.





Now from Conversion tasks window highlight your task and select --> Change



Select --> advance settings drop down


Enable custom file naming -->

1. %showname% S%season%E%episode% 

2. %showname% %airdayshort% %airday% %airmonthshort% %airyear%

I would recommend either way.  TMZ since its a weekday show doesn't record season and episode so date is a better way to say when the recording happened.

File name match --> TMZ*

Comskip INI --> [Locate your configuration file]


OK

OK

If you get the communication error message you will have to start over.  Unknown why this happens.


Select Start on MCEBuddy.


It should locate your TMZ recording(s) and start converting. This process takes a while for good quality video.  It ultimately matters on the recording either HD or SD and HD quality will take longer.


UPDATE 140228;

Had an issue with my operating system HDD not having enough space.  Its a 90GB SSD for speed.  It was reporting only 25GB of space left and American Idol 2hr show failed to convert.

c:\program files\MCEBuddy2x\log\mcebuddy.log

"Error Insufficient disk space using Conversion Task"

If your run into this problem you'll have to move the temporary location where the files can be processed by MCEBuddy to a larger HDD.

I moved mine to a separate HDD from the HDD that stores my Recorded TV (.wtv) files.

Modify here or within MCEBuddy.  g:\MCEBuddy_Temp was my location.


UPDATE 140320;

Posted the configuration files that I have been toying with for approximately a month.  If I update them I'll continue to post the new ones here.

https://drive.google.com/folderview?id=0B2ad8pKdMeSsaGF1aW5HV042WE0&usp=sharing

Saturday, October 26, 2013

Upgrade to a new hard drive on your MacBook Pro and clone OSX and BootCamp



***DISCLAIMER...Its your fault if you screw up your system.  Read through the entire post before attempting.

I purchased a new larger capacity hard disk drive (hdd) to upgrade an existing smaller solid state drive (ssd) because I started running out of room.  I liked the speed of the ssd, dont get me wrong, but a weighed the options and more storage space was a priority.  So I compromised and picked up a Seagate hybrid sshd.  Got one while they were on sale $100 through newegg http://www.newegg.com/Product/Product.aspx?Item=N82E16822178340.

This one took a while to rap my fingers around and I'll say most of it was dumb luck.  I know there is WinClone available on OSX and probably would have made this easier.  Think WinClone is about $30 and honestly probably would have saved me a bunch of time, but being the stubborn ba$T@rd I am, I decide to stick through it on principle.

What I had...
OSX was 10.8
Windows was 8 with media center

Things you'll need:
  • hdd enclosure; this is to connect your existing hdd to copy data over to your new hdd
  • windows 8 installation DVD (or Win7); needed to allow bootcamp creation
  • OSX installation USB or SD drive to boot from; might be able to use the recovery instead
  • windows 7 repair CD; needed to run specific commands to fix the boot (the version of win7 repair CD I used was modified because getting the timing right to engage the CD boot process made me pull more hair out than I care to discuss.  I renamed the bootfix.bin file from the boot directory.  This causes the CD to boot instead of waiting for prompted.  The MacBook keyboard didn't work and the screen picture were out of sync to enable to proper timely to engage the CD.  You can create your own using UltraISO and renaming bootfix.bin under the boot directory.  https://dl.dropboxusercontent.com/u/92763248/Repair%20disc%20Windows%207%2064-bit.iso
  • GPART CD  http://gparted.sourceforge.net/livecd.php
  • Remove encryption.
Pre Steps:

1. With your existing OSX installation, install the gdisk program http://sourceforge.net/projects/gptfdisk/.  We will use that to modify our soon to be screwed up hybrid MBR.  In the mean time lets take a look at a none screwed up hybrid MBR.

Issue the following commands on your old OSX operating system (OS):

sudo fdisk /dev/disk0
sudo gpt -r -vv show disk0
diskutil list 


That will give us a bearing on what your disk use to look like.  Might be handy for later on to have this printed (Command + Shift + 4 than spacebar will get you to screenshot the window).

Steps:
1. Turn off the system and remove the existing hdd and install your new one (Ifixit.com is a pretty great place for these how tos).

2. Attached your old hdd to a USB enclosure and attached to your macbook.  I had purchased a simple USB3 sata link from newegg.com.



3. I made an SD OSX installation media disk. I think you could probably use the recovery hd to do the same thing.  I attached the SD to its slot.

4. I then booted the macbook holding the option key down.


5. It should boot and look like the following.  If the new hdd has no bootable partition you will not see your new drive.  The orange in color USB symbol enclosures are your old OS.  The mountain lion emblem is the OSX installation SD.  This is whether I think its possible to boot from your old OS recovery hd to perform this next action.  I haven't tried that.  The next action will be to copy your old OSX to the new hdd.  I decided to use my OSX SD and booted from that.



6. Select Disk Utility.



7. Your new hdd should be at the top.  Select it on the left pane.



8. Now select the partition drop down.  This is where we are going to partition and format your new drive.  Select 1 partition.



9. Title the new volume "new mac".  Ensure it is Mac OS Extended (Journaled) format.  Then Select options.



10. It needs to be set to GUID Partition Table.  Select OK.  Apply.



11. Partition.



12. Once completed your "new mac" partition should show on the left.  Select the Restore tab.



13. This is were we can copy our existing OSX OS to the new hdd.  Left click on your old OSX partition and hold the mouse button down as you drag it to the Source.  Notice the orange color enclosure icon.  This indicates an external device.  Now drag your newly created "new mac" partition to the Destination.  Should look just like mine. Restore.



14. Erase.




15. Should begin copying the data over.  This processes should take a while.  After that is complete reboot your macbook while holding the option key on boot.  Now boot to your new macbook OS.  I removed the old hdd and installation SD before booting.



16.  Lets see what has changed in our partitions on the new hdd.  Here is an example with the above three commands issued again.  This was on a 1TB HDD.  Notice the size in the fdisk command takes up the entire hdd.


17. We now need to partition the bootcamp slice.  Find the bootcamp icon inside the Utilities directory.
18. Start bootcamp assistant. Continue.

19. If you have a Windows 8 or 7 installation DVD remove the "Create a Windows 7..." check mark.  More than likely since this is about cloning your bootcamp Win8/7 installation you already have bootcamp installed on your Windows OS so remove the check mark.  Install your Win8/7 installation DVD and make sure the "Install Windows 7 or later version" is checked.  Continue.
20. I decide to divide my partitions equally.  You do what you want.  Install.  It is required to have the Win8/7 installation dvd installed to get this step rolling.
 21. You system will automatically reboot to DVD.  ***HOLD THE OPTION KEY DOWN WHEN THE SYSTEM AUTO REBOOTS.*** Now I have done this two different ways.  I've booted to the Windows DVD and formatted the bootcamp partition and not installed a new OS which formats it NTFS.  I've also skipped this step and booted straight to GPART CD as well.  I suspect its not necessary to format because when we use GPART to copy the old Windows OS to our new bootcamp partition it takes on the format of the copied partition anyway so why bother. I ejected the Windows installation DVD and swapped for the GPART CD and continued the boot of the CD.  With GPART booted it should show your new hdd and old hdd so long as both are still attached to your system.  If you forgot just connect and under Gparted select rescan.  Notice the far upper right drop down to select drives.  Now to distinguish which drive is which the /dev/sda is your onboard hdd and /dev/sdb should be your external.  Another way is your new hdd will say BOOTCAMP (FAT32).  Your external or old hdd should say either BOOTCAMP (NTFS) or what ever name you modified the volume name to.  Mine was Win8Pro.


22. Select your external or old hdd.


23. Select your bootcamp partition than the copy button.


24. Now change back to your new hdd and select the FAT32 BOOTCAMP partition.  The paste button should become available and select.



25. It should warn you about this action.  It should be on /dev/sda4. OK.



26. Select Apply.


27. Another warning. Apply.


28. Working.  Could take some time.




29. Once completed, reboot holding the option key and boot to OSX.  The Windows partition shouldn't appear here.  We need to modify the hybrid MBR.  You should see after booting into OSX the Windows drive and its contents though.



30. Now a decided to look at my partitions.  #1 is the EFI boot.  #2 is my OSX.  #3 is my Recovery.  #4 is Windows bootcamp.  Notice the differences in size.  The other thing I noticed was the * was on the OSX #2 and not on the Windows #4 under the fdisk command.  This is where printing out the old hdd setting is nice to see the difference.

new hdd
old hdd

 31. Time to install rEFIt. http://refit.sourceforge.net/  To install right click on the rEFIt.mpkg and open.  You might have to reboot twice for rEFIt to show.  You will notice the Windows boot icon starts to show. 


32. If you try to boot Windows right now you'll get the Recovery blue screen.



33. Reboot into OSX.  Time to make the hybrid MBR right again.  Open terminal.  Enter the following commands and see the images below.  We need to get the hybrid MBR like our old hdd.  This is how to do this.

sudo gdisk /dev/disk0
r
h
2 3 4
y
<enter>
n
<enter>
n
<enter>
y
o
w
y
sudo fdisk /dev/disk0

gdisk = allows us to modify the GUID partition.  Since Windows requires an MBR which GPTs dont use and only have them for legacy purposes we have to fix it.
r = recovery and transformation options
h = make hybrid MBR
2 3 4 (Thats the number 2 space 3 space 4 <enter>) = modifying those three partitions
y = EFI needs to be first
<enter> = select the default because thats the way our original partition was
n = the bootable flag was not set in our original
<enter> = select the default because thats the way our original partition was
n = the bootable flag was not set in our original
<enter> = select the default because thats the way our original partition was
y = because the bootable flag was set in our original
o = print protective MBR data
w = write table to disk and exit
y = confirm
34. Now your systems hybrid MBR is fixed.  Oh but wait there's more. This is where the Win7 Repair CD comes into play.  Again the only thing I modified using UltraISO was the name of the bootfix.bin file.  I renamed it "autoboot_bootfix.bin".  Insert the CD in your MAC and reboot.  Hold option and select the CD to boot from.


35. Windows repair CD should load.  It should also prompt that your Windows install is corrupt.  If you select details it will let you know what its changing.  Select Repair and reboot.  This will be quick so be fast with the option key and attempt to boot Windows now.


36. Thats it.  After booting it might ask to reboot.


37. If your not a fan of rEFIt you can always remove.  http://refit.sourceforge.net/doc/c1s3_remove.html

Hope this works for you...