Friday, August 22, 2014

Viewing H264 raw video files

This is a very useful link I would like to share.

Often one has a video file that one cannot view - maybe from a CCTV system.  It is quite common to be H264 or similar, but does not have a standard wrapper, eg .MOV, or .MP4.

The very good free program VLC does have a feature to allow you to do this.  The following Youtube link does describe how to

As a very quick summary, the following may help

Select
  1.   Tools
  2.   Preferences
  3.   At the bottom select 'All'
  4.   In Stream output  select 'Muxers'
  5.   Then select the  Mux module  - (Automatic is the default)

Monday, August 11, 2014

Companies that promise something they cannot provide

I am not going to name names, but recently I have several reports from my satisfied customers moaning about other companies who have taken their money for software that cannot do what they say.  The main focus is on recovery of video deleted from the camera memory chip.

As I have written here several times, when video is recorded in the camera, the physical sequence of data is different to the logical sequence, the video is written in fragments, but the FAT keeps track of the logical order.  When the file is deleted, this sequence is also lost.  Recovery is complex and cannot be done by simple data carving.

If a video is written to a hard drive, it will normally be sequential and thus a data carving approach to recovery will often work.  I believe that software companies just test hard disk deletions and say it will work.  I have tried a few demo products with my 'favourite' GoPro camera images.  They fall flat on their face.  The software will report files, that look the correct size, but will never ever be viewable.

From CnW Recovery point of view, I have an uphill struggle (on a low budget) to let potentials users know that there is a product that will have a very high success rate of recovery of MP4, MOV, AVI, MTS videos that have been deleted from the camera memory chip.

It is a shame that the internet has to boast and boast even when something does not work.  At CnW, not everything works, but there is always free backup to resolve the problem - fairly quickly.

Tuesday, April 8, 2014

Apple disk that was a FAT32

I see a fair number of Apple disks and most start with sector 0 having the letter 'ER' in the first two bytes.  Others have a GPT boot sector.  They then always have an HFS+ file system.

I have just seen a new variation.  It started with a normal ER boot sector.  The next sector started, as normal with PM and was the Apple Partition map pointers.  The third sector then very unusually pointed to DOS_FAT_32.

First attempt to read the disk with CnW failed, but with a few manual inputs it was possible.  Firs I had to set the partition type to be FAT32, and the partition start to be 0x40.  After this, the disk read OK with CnW Recovery software.  Due to hardware issues (the reason I had the drive in the first place) I could not try the drive on a MAC, so I do not know if a standard MAC would read the drive or not.

A future version of CnW will understand this type of drive automatically.

Wednesday, January 29, 2014

Deleted GoPro video with high and low resolution

MP4 video is often recorded out of sequence.  This is because when the recording starts there is no knowledge of the file size.  Thus the meta data is added physically at the end of the physical file, though logically it is stored near the file start.

CnW can process this type of deleted file.  A new variation found in one version of the GoPro Hero 3+ Black camera is stored both high and low resolution video.  In this case, the video is multiplexed so physically on the disk there will be sections of high resolution, followed by low resolution.  The programming challenge is to determine which cluster is part of which video stream.  By examining the meta data this is possible and current results are very encouraging.  The next stage will be to ensure there are no glitches in the recovered streams

 www.cnwrecovery.com

Sunday, December 29, 2013

Recover deleted MP4 files from Sony PMW-F3 update

My last post reported good progress on recovery of Sony PMW-F3 files.  I am now pleased to report some extremely good results.

The final approach involved scanning the complete disk for possible audio and video elements.  As this is fairly machine intensive it was very satisfying from the programming point of view to make use of parallel processing.  While one cluster was being read from the physical device, the previous cluster was examined for audio data, and in parallel, video data.

Once the disk has been scanned it is then quick to isolate any pre scanned video structure with it's location within a cluster.  On a small memory device one is normally luck that each structure will have a unique offset.  For large memory device there will be duplicate entries.  In these cases the physical location of the matching cluster will be used to determine the correct one.

When a video file is deleted it is normal for the data to remain, although may well be non sequential.  The critical section to find is the moov atom.  This contains pointers to frame starts and audio buffers.  By tying up the moov pointers with the physical clusters a valid file can be reconstructed.

Simple?  In some respects yes, but recognising patterns is not a simple computer task.  It is very much an exercise in fitting together the best matches.  It is a bit like doing a jigsaw that has no picture.  When it works, the results are fantastic.

Many videos are recorded in two sections, the Sony sometimes uses over 100 sections, so joining these together is a great success.

Sunday, December 1, 2013

Sony Video Camera PMW-F3 deleted MP4 recovery

A major and on going CnW Recovery development is processing fragmented video files.  In particular these are ones that have been deleted on the camera.  When a FAT32 file is deleted, the allocation table is cleared down and so the order of clusters is lost.  For may applications, this does not matter as the file is sequential and so can be recovered - and this is often true for photos.  Videos however tend to be long and an MP4 /3GP / MOV file has three main sections (or atoms).

  • A short 'ftyp' header
  • A mid size 'moov' that stores pointers
  • A big 'mdat' that stores the audio and video
Logically the sequence is normally either ftyp-moov-mdat  or ftyp-mdat-moov.

When recording it is impossible to know the length of the moov or mdat atom.  For this reason the mdat is stored on the memory chip, and often the moov is stored in the camera RAM until the end.  On finalisation it is then written to the memory chip.  If the required sequence is ftyp-moov-mdata some 'clever' fiddling of the FAT is performed by the camera to make the file logically sequential.

The CnW program has been developed to handle the above for many video types but then  the Sony PMW-F3 format was found.  The big difference this time is that the 'mdat' is stored in numerous chunks, and not always in sequence on the memory chip.  The challenge that CnW is working on is to find these chunks and reconstruct the video.  This is performed in several stages
  • Chip is scanned for all ftyp, mdat and moov headers
  • Chip is scanned for all MP4A audio clusters
  • If required a fragmented moov is reconstructed
  • The video and audio frames are located based on offset within a cluster.  Special routines are required when there are multiple audio or video frames stored at the same location within a cluster
  • Often the frame pointers point to clusters several clusters later than the previous one. In these cases the gap between the known cluster locations has be filled in by working forward and backwards from known good locations
The current results are very reasonable, but still more tweaking is required.  However one customer reported that CnW recovery does recover viewable video, and no other program got anywhere near this. 

CnW is very sceptical of many adverts that claim video recovery - it may work from a hard disk, but CnW has major doubts about working form camera memory chips

Monday, September 16, 2013

GoPro Hero 3 recovery of deleted files

I have often said of data structures, 'if it can be done, it will be done'.  ie almost anything is possible, and so expect many variations.  In the words of the 'Hitch Hikers Guide to the Universe' - Expect the unexpected.

A recent view of a GoPro Hero 3 camera confirmed the above.  As I have written in the past, video camera often record the data not in a different physical to logical sequence.  The slight of hand is that the FAT defines the logical sequence.  Thus many camera record the video data physically earlier than the file start sector.  On the GoPro Hero 3 camera another twist has been discovered.  Two video streams can be recorded at the same time - a high resolution, and a smaller low resolution.  The physical sequence on the disk could be as below



Just to complicate the above, there can also be jpeg thumbnails, and text status files inserted in the above stream.  Standard data carving is totally useless as nothing is in sequence.

The CnW MP4/3GP Wizard works hard, but will recover, with a high success rate videos that have been deleted.  www.cnwrecovery.com/html/mp4_wizard.html

Tuesday, September 3, 2013

Windows 8

It was time for a new PC and so Windows 8 - 64 seems the obvious option in order to keep up to date.  I ordered reasonable spec system (3.6GHz, Core 7, 12GB RAM) and a week or so later it arrived.

I have had Windows 8-32 since it came out, but not as a main PC.  This post will describe some of the issues I've had, and some solutions.

My work (data recovery) requires lots of disk space.  I keep the main 2TB system drive as the system and development drive.  All customer data is stored on other drives - or a new 9TB NAS RAID.  My first job was to add a second 3TB drive to the PC.  This was a reasonable quick job with the screwdriver, and the drive was seen in the BIOS.  However, what ever I did, I could not see it in Windows 8.  Google to the rescue and it turns out that Windows 8, being much more secure will not recognise extra hardware (though it could see USB drives).  The solution was a BIOS option switch to remove the 'Secure boot' option.  Drive now seen, and the secure boot can now be re-enabled.

The next problem was Norton antivirus.  The PC came with McAfee, but the rest of my systems all have Norton, on a group licence.  I uninstall McAfee and installed Norton 360 but then problems began.  Various problems, mainly IE10 and other internet related products had issues.  Worse still, each time I rebooted the PC, Norton stopped working and often came up with the error 8504,101.  Google had several ideas for this, including running NPE.exe, uninstall and re-install.  I tried many of these suggestions, but none worked.  The error message is common on Google posts, and so I installed AVG and started to get some work done.  Even the Beta version of N360 had the same problem.

On hardware, I have become a complete fan of multiple screens.  The new PC has 3 video outputs,  but 2 of them are HDMI, and I only have (old) VGA screens.  I hope the eBay adaptor will arrive soon and let me work with 2, or maybe 3 screens at once.

A final issue I had was over a security dongle.  The drive is not signed, and so Windows 8 will not load it.  The solution was the result of another search and involves booting Windows in a mode to accept unsigned drivers.  The instructions (with due credit to someone else) are as follows

1. Windows Key + R
2. Enter shutdown.exe /r /o /f /t 00
3. Click the OK button
System reboots here

4. System will restart to a Choose an option screen
5. Select Troubleshoot from Choose an option screen
6. Select Advanced options from Troubleshoot screen
7. Select Windows Startup Settings from Advanced options screen
8. Click Restart button
9. System will restart to Advanced Boot Options screen
10. Select Disable Driver Signature Enforcement
11. Once the system starts, install the drivers as normal
 
 

Conclusion

Looking forward to Windows 8.1 where I hope the metro screen will be less important, though I will start playing with some of the apps.


Saturday, January 5, 2013

Encrypted drives

I was recently helping a potential customer with a data recovery problem.  The problem was a Western Digital external drive (1TB) that had a damaged USB connector on the case.  The drive was removed and placed in a USB caddy (for common practise).  The drive appears physically OK, but very few files could be read.

A few scans and logs were transfered and it appears that there were files at the end of the disk, but the middle area was almost totally blank, ie very few file signatures recognised.  The next stage was a a disk scan (a CnW Forensic feature) which scans the complete drive and shows the broad category of data in each sector.  This includes text, blank, directory entry and compressed.  Most of the middle was deteceted as compressed.  A compressed sector (in this logic) is one with many different byte values in the sector and will detect Zip files, JPEGs, MPEGS, music files as well as encrypted data.

The customer had not used encryption, or a program such as TrueCrypt so the results did not make much sense.  However, the more I thought about it, the more the data looked as the disk was compressed.

A bit of Google research did reveal something I was not expecting.  WD do make external drives with built in compression, controlled by the internal controller board.  This is enabled, even if a password is not entered and could explain the situation with this drive.  The only solution is to read the drive with the original controller board.  Otherwise, to read the encrypted data can vary between very difficult to impossible.

This configuration came a surprise for me, and I must now be aware that the dirve case may be important, and not just accept the naked drive.

Saturday, October 27, 2012

Other data recovery companies

It is not good british policy to 'throw' mud at other businesses, but in the past week I have come across two cases of what best be described as incompetence.

The first case was a company trying to recover photos from a damaged or corrupted sD card.  At the end of the recovery the results were poor.  Then for some reason they returned the poor results on the same memory card, so overwriting the files and making the chance of future recovery extremely remote.

Another job I received a partially deconstructed CE card.  The company was about to remove the chips, but realised they were not compatible with their memory chip reading equipment, having an unusual hidden pin layout.  What I cannot understand is that the card would read OK as a complete card, but just had a few sectors not working.  OK these were part of the FAT but actually not at all critical.  It was possible to read the card, and using CnW recovery software a full recovery of the critical files was made.

The biggest golden rule with data recovery is to never change data on the problem device.  When ever possible it is best to connect the device using a write blocker as this will also prevent a PC doing a virus erase, defrag, or any other device the PC may decide to do which is normally invisible and harmless.  In the second case, the issue was solved with software as only a few sectors had failed.  Chip removal may well have come across the same failed sectors and there is always the chance of more damage when subjecting memory chips with lots of hot air, and physical strain.