Railway track fasteners are used to fasten the railway track onto the crosstie and improve the train’s stability and safety. Automatic detection systems have been developed for track safety. Most of these systems deployed line scan sensors to capture high-quality track images. These sensors can capture high-resolution images, but they are also expensive. In addition, the recognition kernels range from traditional computer vision to deep learning methods. In this study, we set up a track fastener sensing device on a flat track car by using general sport cameras and LED lamps to capture images of track fasteners. You Only Look Once (Yolo) v3 is also used instead of earlier convolution neural networks (CNNs) for defect inspection. A cloud server is built for users to queue their captured fastener videos to the first buffer for upload, and uploaded videos can be queued to a second buffer for defective track fastener detection. The trained Yolo v3 neural network classification module is encapsulated as a web application interface (API) for performing the task. In experiments, track fastener videos along a total of 70 km of track were captured with a resolution of 1920 × 1080 at a speed of up to 35 km/h. Six normal and four defective fastener types were defined for inspection. We split the dataset into 80% for training and 20% for testing. The average precision rates for normal and defective fasteners were 83 and 89%, respectively. Finally, the coordinates of defective fasteners were interpolated from GPS positions recorded by a sport camera. The nearest hectometer stake and the offset of each defective fastener were calculated to assist track workers to find the defective fasteners and fix them.