Building Billscan AI: Edge OCR and Real-Time Expense Parsing with SwiftUI and VisionKit
Published on 2026-10-25 · 7 min read · by Kreggscode
Building Billscan AI: Edge OCR and Real-Time Expense Parsing with SwiftUI and VisionKit
Tracking everyday receipts and corporate expenses has historically been a tedious, manual chore. Traditional scanning apps either rely on cumbersome server-side OCR pipelines—introducing privacy concerns and high latency—or suffer from poor optical recognition under skewed angles and crumpled thermal paper.
To solve this, we engineered [Billscan AI](https://kreggscode.com/apps/billscan-ai/), a high-performance native iOS receipt scanner built entirely on SwiftUI 6, VisionKit, and on-device CoreML. Billscan AI processes paper receipts, invoices, and digital bills in under 200 milliseconds directly on the iPhone's Neural Engine.
You can download the live production app on the Apple App Store here:
👉 [Download Billscan AI on the Apple App Store](https://apps.apple.com/app/id6773950473)
Explore the developer's full catalog on the [Subodh Godi Apple Developer Portfolio](https://apps.apple.com/developer/subodh-godi/id1831370968).
---
1. The Architectural Challenge: Edge vs. Cloud OCR
Cloud-based OCR APIs (such as AWS Textract or Google Cloud Vision) present severe architectural liabilities for mobile utility apps:
1. Latency Penalty: Network round-trip times hover between 1.5s to 3.5s per image upload.
2. Privacy Risks: PII, credit card merchant numbers, and sensitive corporate expenditures are transmitted over public networks.
3. Operational Overhead: Cloud OCR charges $1.50 per 1,000 pages, rapidly eroding app unit economics.
By shifting 100% of the computer vision and extraction pipeline directly onto the device, Billscan AI eliminates cloud costs while guaranteeing strict user privacy.
┌────────────────────────────────────────────────────────┐
│ Billscan AI Optical Pipeline │
└────────────────────────────────────────────────────────┘
│
▼
┌──────────────────────────────────┐
│ DataScannerViewController │
│ (VisionKit Edge Document Grab) │
└────────────────┬─────────────────┘
│ Raw Frame Buffer
▼
┌──────────────────────────────────┐
│ VNRecognizeTextRequest │
│ (Accurate Mode + Language Hints)│
└────────────────┬─────────────────┘
│ Bounding Box + Text Streams
▼
┌──────────────────────────────────┐
│ Spatial Geometry Parser │
│ (Line-item & Total Correlation) │
└────────────────┬─────────────────┘
│ Structured Transaction Model
▼
┌──────────────────────────────────┐
│ SwiftData Encrypted Persistence │
│ (Multi-Currency Normalization) │
└──────────────────────────────────┘---
2. Implementing VisionKit Sub-Second Edge Scanning
In iOS 16 and Swift 6, Apple introduced DataScannerViewController, which streams camera buffers directly through the Neural Engine. Here is how we configure high-accuracy text extraction with zero dropped UI frames:
import SwiftUI
import Vision
import VisionKit
@MainActor
final class ReceiptScannerEngine: ObservableObject {
@Published var detectedTotal: Decimal?
@Published var merchantName: String = ""
@Published var lineItems: [ReceiptItem] = []
private let textRecognitionRequest = VNRecognizeTextRequest()
init() {
configureEngine()
}
private func configureEngine() {
textRecognitionRequest.recognitionLevel = .accurate
textRecognitionRequest.usesLanguageCorrection = true
textRecognitionRequest.recognitionLanguages = ["en-US", "es-ES", "fr-FR", "de-DE"]
}
func processFrame(pixelBuffer: CVPixelBuffer) async throws {
let requestHandler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: .up)
try await Task.detached(priority: .userInitiated) { [request = self.textRecognitionRequest] in
try requestHandler.perform([request])
guard let observations = request.results else { return }
// Correlate bounding boxes spatially
let extractedData = SpatialReceiptParser.parse(observations)
await MainActor.run {
self.detectedTotal = extractedData.total
self.merchantName = extractedData.merchant
self.lineItems = extractedData.items
}
}.value
}
}
---
3. Spatial Geometry Parsing: Extracting Merchant, Tax, and Totals
A common failure mode in naive OCR apps is sorting words top-to-bottom. On receipts, prices sit horizontally to the right of descriptions, often across wide whitespace gaps.
Billscan AI utilizes a 2D Spatial Proximity Algorithm:
1. Words with overlapping vertical coordinates ($Y_{min} \dots Y_{max}$) are grouped into horizontal text bands.
2. The leftmost text band is classified as the Merchant Descriptor.
3. Currency symbols ($, €, £, ¥) and decimal figures on the far right edge of the horizontal band are matched as Amounts.
4. The highest confidence monetary value at the bottom third of the bounding container is extracted as the Total Due.
---
4. Key Takeaways and Experience It Live
Ready to streamline your financial bookkeeping?