Folder import\sync rework (#18)

* feat: rework Folder Sync architecture to separate Managed vs Linked books

- Introduced "Managed" (imported to app storage) vs "Linked" (SAF URI) book logic.
- Disabled Google Drive/Metadata synchronization for Folder-Linked books to respect privacy and storage.
- Updated deletion logic: Folder books are now marked as deleted in the DB to blacklist them from future auto-scans, while Managed books are permanently purged from local and cloud storage.
- Enhanced folder sync discoverability by adding a direct "Sync Folder" navigation button in Home screen.
- Implemented reactive pager navigation in MainScreen and LibraryScreen via LaunchedEffect to allow programmatic tab switching from the ViewModel.

* Implemented a local folder synchronization system to track and update book metadata across devices.

Key changes:
- Added `FolderBookMetadata` and `LocalSyncUtils` for serializing and managing metadata files in a `.episteme` directory.
- Implemented bidirectional sync in `FolderSyncWorker` to reconcile local database state with folder-based metadata using timestamps and Syncthing conflict resolution.
- Updated `MainViewModel` to trigger folder metadata sync when closing a book and verify for remote updates upon opening.
- Enhanced folder management with manual scan support via `WorkManager` and automatic cleanup of database entries when a folder is disconnected.
- Added `RecentFileDao` methods to support bulk deletion and prefix-based file lookups.

* Removed API 34 restriction for chapter processing and updated local folder sync directory name

* Increased TTS timeouts, added edge-to-edge support in MainScreen, and improved TtsChunk mapping safety

* Improved folder synchronization and metadata management.

- Added metadata-only sync option and triggered it on app start.
- Implemented hidden metadata filenames (prefixed with `.`) to reduce file clutter.
- Added automatic creation of `.nomedia` files in the sync directory.
- Enhanced sync conflict resolution by picking the latest metadata and cleaning up obsolete or orphaned files.
- Added "Sync Metadata" button to the Library screen.
- Updated UI labels for local folder sync to clarify Google Drive integration.
- Refactored `FolderSyncWorker` to support targeted metadata-only synchronization.

* Implement folder sync migration and refactoring

- Added a migration dialog to inform users about the folder sync refactor, where books are now read directly from external storage.
- Updated `MainViewModel` to handle migration state and trigger a full scan upon completion or dismissal of the dialog.
- Modified `FolderSyncWorker` to support legacy book matching during migration, updating file URIs and cleaning up internal app storage for migrated books.
- Improved synchronization logic between local and remote metadata, including cleanup of orphaned metadata files.
- Refined `RecentFileItem` updates during sync to preserve progress and bookmarks based on last modified timestamps.

* Implemented pull-to-refresh for library sync and enhanced folder file deletion.

- Added `isRefreshing` state to `MainViewModel` and integrated `PullToRefreshBox` in `HomeScreen`.
- Implemented `refreshLibrary` to trigger cloud and folder metadata synchronization.
- Updated deletion logic to physically remove files and metadata from synced local folders.
- Added a warning to the delete confirmation dialog when removing folder-synced items.
- Improved folder sync reliability by performing lazy cleanup of missing files during interaction and sync.
- Fixed a bug where sync loading indicators would not retract on failure or cancellation.

* Improved bookmark navigation and scroll synchronization in EPUB reader

- Refined bookmark navigation logic for vertical scroll mode to handle chunk injection more reliably.
- Added `isNavigatingToBookmark` state to show a loading overlay during long jumps.
- Implemented `onScrollFinished` callback in `CfiJsBridge` to synchronize UI state with WebView scroll completion.
- Updated `ChapterWebView` to use `rememberUpdatedState` for JavaScript bridge callbacks to ensure data consistency.
- Improved `getCurrentCfi` and `scrollToCfi` in `epub_reader.js` with better visibility probing and retry logic for detached nodes.

* Implemented background metadata extraction for folder sync.

Key changes:
- Added `MetadataExtractionWorker` to handle heavy metadata and cover extraction for files in the background.
- Refactored `FolderSyncWorker` to perform fast file discovery using placeholders, enqueuing the metadata worker upon completion.
- Added `getFolderBooksWithoutCovers` query to `RecentFileDao`.
- Updated UI components (`EmptyState`, `MainViewModel`) to improve user feedback during sync and provide clear setup actions.
- Enhanced `EmptyState` composable to support secondary actions and custom button text.

* Updated Library Screen UI
This commit is contained in:
Aryan 2026-03-01 22:57:53 +05:30 committed by GitHub
parent 60f6566b31
commit a81df6921d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
21 changed files with 3378 additions and 2478 deletions

View file

@ -0,0 +1,102 @@
// FolderBookMetadata.kt
package com.aryan.reader.data
import com.aryan.reader.FileType
import org.json.JSONObject
data class FolderBookMetadata(
val bookId: String,
val title: String?,
val author: String?,
val displayName: String,
val type: String,
val lastChapterIndex: Int?,
val lastPage: Int?,
val lastPositionCfi: String?,
val progressPercentage: Float,
val isRecent: Boolean,
// REMOVED: val isDeleted: Boolean,
val lastModifiedTimestamp: Long,
val bookmarksJson: String?,
val locatorBlockIndex: Int?,
val locatorCharOffset: Int?
) {
fun toJsonString(): String {
val json = JSONObject()
json.put("bookId", bookId)
json.put("title", title)
json.put("author", author)
json.put("displayName", displayName)
json.put("type", type)
json.put("lastChapterIndex", lastChapterIndex ?: -1)
json.put("lastPage", lastPage ?: -1)
json.put("lastPositionCfi", lastPositionCfi)
json.put("progressPercentage", progressPercentage.toDouble())
json.put("isRecent", isRecent)
// REMOVED: json.put("isDeleted", isDeleted)
json.put("lastModifiedTimestamp", lastModifiedTimestamp)
json.put("bookmarksJson", bookmarksJson)
json.put("locatorBlockIndex", locatorBlockIndex ?: -1)
json.put("locatorCharOffset", locatorCharOffset ?: -1)
return json.toString()
}
companion object {
fun fromJsonString(jsonString: String): FolderBookMetadata {
val json = JSONObject(jsonString)
fun JSONObject.optStringNull(key: String): String? {
return if (has(key) && !isNull(key)) getString(key) else null
}
fun JSONObject.optIntNull(key: String): Int? {
val value = optInt(key, -1)
return if (value == -1) null else value
}
return FolderBookMetadata(
bookId = json.getString("bookId"),
title = json.optStringNull("title"),
author = json.optStringNull("author"),
displayName = json.optString("displayName", "Unknown"),
type = json.optString("type", "PDF"),
lastChapterIndex = json.optIntNull("lastChapterIndex"),
lastPage = json.optIntNull("lastPage"),
lastPositionCfi = json.optStringNull("lastPositionCfi"),
progressPercentage = json.optDouble("progressPercentage", 0.0).toFloat(),
isRecent = json.optBoolean("isRecent", true),
// REMOVED: isDeleted deserialization
lastModifiedTimestamp = json.optLong("lastModifiedTimestamp", 0L),
bookmarksJson = json.optStringNull("bookmarksJson"),
locatorBlockIndex = json.optIntNull("locatorBlockIndex"),
locatorCharOffset = json.optIntNull("locatorCharOffset")
)
}
}
}
// Update the converter
fun FolderBookMetadata.toRecentFileItem(uriString: String?, coverPath: String?, sourceFolderUri: String?): RecentFileItem {
return RecentFileItem(
bookId = this.bookId,
uriString = uriString,
type = try { FileType.valueOf(this.type) } catch (_: Exception) { FileType.EPUB },
displayName = this.displayName,
timestamp = System.currentTimeMillis(),
coverImagePath = coverPath,
title = this.title,
author = this.author,
lastChapterIndex = this.lastChapterIndex,
lastPage = this.lastPage,
lastPositionCfi = this.lastPositionCfi,
locatorBlockIndex = this.locatorBlockIndex,
locatorCharOffset = this.locatorCharOffset,
progressPercentage = this.progressPercentage,
isRecent = this.isRecent,
isAvailable = true,
lastModifiedTimestamp = this.lastModifiedTimestamp,
isDeleted = false, // ALWAYS FALSE for folder sync
bookmarksJson = this.bookmarksJson,
sourceFolderUri = sourceFolderUri
)
}

View file

@ -0,0 +1,260 @@
// LocalSyncUtils.kt
package com.aryan.reader.data
import android.content.Context
import android.net.Uri
import androidx.documentfile.provider.DocumentFile
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import timber.log.Timber
object LocalSyncUtils {
private const val SYNC_DIR_NAME = "episteme"
private const val TAG = "FolderSync"
suspend fun saveMetadataToFolder(
context: Context,
sourceFolderUri: Uri,
metadata: FolderBookMetadata
) = withContext(Dispatchers.IO) {
try {
val rootTree = DocumentFile.fromTreeUri(context, sourceFolderUri) ?: return@withContext
val syncDir = getOrCreateSyncDir(rootTree)
if (syncDir == null) {
Timber.tag(TAG).e("Could not create/find $SYNC_DIR_NAME directory in $sourceFolderUri")
return@withContext
}
// Ensure .nomedia exists to prevent gallery clutter
ensureNoMedia(syncDir)
// Use hidden filename to avoid "Recents" clutter
val hiddenFileName = ".${metadata.bookId}.json"
val legacyFileName = "${metadata.bookId}.json"
// Check for existing files (Hidden OR Legacy)
val existingHidden = syncDir.findFile(hiddenFileName)
val existingLegacy = syncDir.findFile(legacyFileName)
// Prefer hidden, fallback to legacy for conflict check
val existingFile = existingHidden ?: existingLegacy
if (existingFile != null && existingFile.exists()) {
try {
val existingContent = context.contentResolver.openInputStream(existingFile.uri)?.use { input ->
input.bufferedReader().use { it.readText() }
}
if (existingContent != null) {
val existingMeta = FolderBookMetadata.fromJsonString(existingContent)
val diff = existingMeta.lastModifiedTimestamp - metadata.lastModifiedTimestamp
// Clobber Protection
if (existingMeta.lastModifiedTimestamp > metadata.lastModifiedTimestamp) {
Timber.tag(TAG).w("ClobberCheck: ABORTING save for ${metadata.bookId}. Folder has newer data.")
return@withContext
}
}
} catch (e: Exception) {
Timber.tag(TAG).e(e, "Failed to read existing metadata for conflict check")
}
// Delete the existing file (whether hidden or legacy) before writing new one
try {
existingFile.delete()
} catch (e: Exception) {
Timber.tag(TAG).w("Failed to delete existing metadata file: ${e.message}")
}
}
// If we had a legacy file that wasn't the 'existingFile' (edge case), delete it too
if (existingLegacy != null && existingLegacy.exists()) {
try { existingLegacy.delete() } catch (_: Exception) {}
}
val newFile = syncDir.createFile("application/json", hiddenFileName)
if (newFile == null) {
Timber.tag(TAG).e("Could not create metadata file for ${metadata.bookId}")
return@withContext
}
val jsonString = metadata.toJsonString()
try {
context.contentResolver.openOutputStream(newFile.uri)?.use { output ->
output.write(jsonString.toByteArray())
}
Timber.tag(TAG).d("Saved metadata for ${metadata.bookId} (Hidden)")
} catch (e: Exception) {
Timber.tag(TAG).e(e, "Failed to write content to metadata file for ${metadata.bookId}")
try { newFile.delete() } catch (_: Exception) {}
}
} catch (e: Exception) {
Timber.tag(TAG).e(e, "Failed to save local metadata to folder.")
}
}
suspend fun getBookMetadata(
context: Context,
sourceFolderUri: Uri,
bookId: String
): FolderBookMetadata? = withContext(Dispatchers.IO) {
try {
val rootTree = DocumentFile.fromTreeUri(context, sourceFolderUri) ?: return@withContext null
val syncDir = findSyncDir(rootTree) ?: return@withContext null
// Find all related files: hidden, legacy, and conflicts
val relatedFiles = syncDir.listFiles().filter { file ->
val name = file.name ?: ""
// Match: .bookId.json, bookId.json, or containing .sync-conflict
(name.contains(bookId)) && (name.endsWith(".json") || name.contains(".sync-conflict"))
}
if (relatedFiles.isEmpty()) return@withContext null
return@withContext resolveAndCleanConflicts(context, relatedFiles, bookId)
} catch (e: Exception) {
Timber.tag(TAG).e(e, "Error resolving book metadata for $bookId")
}
return@withContext null
}
/**
* Reads all candidate files, picks the winner (highest timestamp),
* and deletes the losers (cleanup).
*/
private fun resolveAndCleanConflicts(
context: Context,
files: List<DocumentFile>,
bookId: String
): FolderBookMetadata? {
var bestMeta: FolderBookMetadata? = null
var bestFile: DocumentFile? = null
// 1. Find the winner
files.forEach { file ->
try {
val jsonString = context.contentResolver.openInputStream(file.uri)?.use { input ->
input.bufferedReader().use { it.readText() }
}
if (jsonString != null) {
val meta = FolderBookMetadata.fromJsonString(jsonString)
// Ensure this file actually belongs to the book (defensive check against partial name matches)
if (meta.bookId == bookId) {
if (bestMeta == null || meta.lastModifiedTimestamp > bestMeta!!.lastModifiedTimestamp) {
bestMeta = meta
bestFile = file
}
}
}
} catch (e: Exception) {
Timber.tag(TAG).e(e, "Failed to parse conflict file: ${file.name}")
}
}
// 2. Clean up losers
if (bestMeta != null && bestFile != null) {
val filesToDelete = files.filter { it.uri != bestFile!!.uri }
if (filesToDelete.isNotEmpty()) {
Timber.tag(TAG).i("Resolving conflicts for $bookId. Winner: ${bestFile!!.name}. Deleting ${filesToDelete.size} obsolete files.")
filesToDelete.forEach {
try { it.delete() } catch(_: Exception) {}
}
}
// 3. Migrate Legacy to Hidden if needed
val winnerName = bestFile!!.name ?: ""
if (!winnerName.startsWith(".")) {
Timber.tag(TAG).i("Migrating legacy file to hidden: $winnerName")
// We can't always rename easily with DocumentFile, so we allow 'saveMetadataToFolder'
// to handle the actual file swap next time a write happens, OR we could force a rewrite.
// For now, we leave it. The clutter is reduced by deleting conflicts.
// The next save operation will create the hidden file and delete this one.
}
}
return bestMeta
}
suspend fun getAllFolderMetadata(
context: Context,
sourceFolderUri: Uri
): Map<String, FolderBookMetadata> = withContext(Dispatchers.IO) {
val finalResults = mutableMapOf<String, FolderBookMetadata>()
try {
val rootTree = DocumentFile.fromTreeUri(context, sourceFolderUri) ?: return@withContext finalResults
val syncDir = findSyncDir(rootTree) ?: return@withContext finalResults
// Ensure .nomedia exists while scanning
ensureNoMedia(syncDir)
val allFiles = syncDir.listFiles()
// Group files by bookId.
// Filename formats:
// 1. hidden: .[bookId].json
// 2. legacy: [bookId].json
// 3. conflict: .[bookId].sync-conflict... or [bookId].sync-conflict...
val groupedFiles = allFiles
.filter { it.name?.endsWith(".json") == true || it.name?.contains(".sync-conflict") == true }
.groupBy { file ->
var name = file.name ?: ""
// Remove leading dot
if (name.startsWith(".")) name = name.substring(1)
// Remove conflict suffix
name = name.substringBefore(".sync-conflict")
// Remove extension
name.substringBefore(".json")
}
groupedFiles.forEach { (bookId, files) ->
val winner = resolveAndCleanConflicts(context, files, bookId)
if (winner != null) {
finalResults[bookId] = winner
}
}
Timber.tag(TAG).d("getAllFolderMetadata: Consolidated ${groupedFiles.size} book records.")
} catch (e: Exception) {
Timber.tag(TAG).e(e, "Error scanning .episteme folder")
}
return@withContext finalResults
}
private fun findSyncDir(root: DocumentFile): DocumentFile? {
// Look for exact match first
val standardDir = root.findFile(SYNC_DIR_NAME)
if (standardDir != null && standardDir.isDirectory) return standardDir
// Fallback search
val files = root.listFiles()
return files.firstOrNull {
it.isDirectory && (it.name == SYNC_DIR_NAME)
}
}
private fun getOrCreateSyncDir(root: DocumentFile): DocumentFile? {
val existing = findSyncDir(root)
if (existing != null) return existing
return root.createDirectory(SYNC_DIR_NAME)
}
private fun ensureNoMedia(dir: DocumentFile) {
if (dir.findFile(".nomedia") == null) {
try {
dir.createFile("application/octet-stream", ".nomedia")
} catch (e: Exception) {
Timber.tag(TAG).w("Failed to create .nomedia file")
}
}
}
}

View file

@ -17,6 +17,7 @@
*
* mail: epistemereader@gmail.com
*/
// RecentFileDao.kt
package com.aryan.reader.data
import androidx.room.Dao
@ -33,6 +34,9 @@ interface RecentFileDao {
@Query("SELECT * FROM recent_files WHERE isDeleted = 0 ORDER BY timestamp DESC")
fun getRecentFiles(): Flow<List<RecentFileEntity>>
@Query("SELECT * FROM recent_files WHERE sourceFolderUri = :sourceFolderUri AND isDeleted = 0")
suspend fun getFilesBySourceFolder(sourceFolderUri: String): List<RecentFileEntity>
@Query("SELECT * FROM recent_files")
suspend fun getAllFiles(): List<RecentFileEntity>
@ -57,6 +61,12 @@ interface RecentFileDao {
@Query("SELECT * FROM recent_files WHERE uriString = :uriString")
suspend fun getFileByUri(uriString: String): RecentFileEntity?
@Query("DELETE FROM recent_files WHERE sourceFolderUri = :sourceFolderUri")
suspend fun deleteFilesBySourceFolder(sourceFolderUri: String)
@Query("SELECT * FROM recent_files WHERE bookId LIKE :prefix || '%'")
suspend fun getFilesWithIdPrefix(prefix: String): List<RecentFileEntity>
@Query("DELETE FROM recent_files")
suspend fun clearAll()
@ -74,4 +84,7 @@ interface RecentFileDao {
@Query("UPDATE recent_files SET isRecent = 0, lastModifiedTimestamp = :timestamp WHERE bookId IN (:bookIds)")
suspend fun markAsNotRecent(bookIds: List<String>, timestamp: Long)
@Query("SELECT * FROM recent_files WHERE sourceFolderUri IS NOT NULL AND coverImagePath IS NULL AND isDeleted = 0")
suspend fun getFolderBooksWithoutCovers(): List<RecentFileEntity>
}

View file

@ -17,11 +17,13 @@
*
* mail: epistemereader@gmail.com
*/
// RecentFilesRepository.kt
package com.aryan.reader.data
import android.content.Context
import android.graphics.Bitmap
import android.net.Uri
import androidx.core.net.toUri
import timber.log.Timber
import com.aryan.reader.BookImporter
import com.aryan.reader.paginatedreader.Locator
@ -34,7 +36,7 @@ import java.io.FileOutputStream
private const val COVER_CACHE_DIR = "cover_cache"
class RecentFilesRepository(context: Context) {
class RecentFilesRepository(private val context: Context) {
private val recentFileDao = AppDatabase.getDatabase(context).recentFileDao()
private val coverCacheDir = File(context.filesDir, COVER_CACHE_DIR)
@ -60,6 +62,10 @@ class RecentFilesRepository(context: Context) {
return@withContext recentFileDao.getFileByUri(uriString)?.toRecentFileItem()
}
suspend fun getFilesBySourceFolder(sourceFolderUri: String): List<RecentFileItem> = withContext(Dispatchers.IO) {
return@withContext recentFileDao.getFilesBySourceFolder(sourceFolderUri).map { it.toRecentFileItem() }
}
suspend fun getAllFilesForSync(): List<RecentFileItem> = withContext(Dispatchers.IO) {
return@withContext recentFileDao.getAllFiles().map { it.toRecentFileItem() }
}
@ -109,6 +115,42 @@ class RecentFilesRepository(context: Context) {
Timber.d("Added/Updated recent file in DB: ${item.displayName}")
}
suspend fun syncLocalMetadataToFolder(bookId: String) = withContext(Dispatchers.IO) {
val entity = recentFileDao.getFileByBookId(bookId) ?: return@withContext
val folderUriString = entity.sourceFolderUri
if (folderUriString != null) {
Timber.d("Syncing metadata to local folder for book: $bookId")
val metadata = FolderBookMetadata(
bookId = entity.bookId,
title = entity.title,
author = entity.author,
displayName = entity.displayName,
type = entity.type.name,
lastChapterIndex = entity.lastChapterIndex,
lastPage = entity.lastPage,
lastPositionCfi = entity.lastPositionCfi,
progressPercentage = entity.progressPercentage ?: 0f,
isRecent = entity.isRecent,
lastModifiedTimestamp = entity.lastModifiedTimestamp,
bookmarksJson = entity.bookmarks,
locatorBlockIndex = entity.locatorBlockIndex,
locatorCharOffset = entity.locatorCharOffset
)
LocalSyncUtils.saveMetadataToFolder(
context = context, // Now correctly references the property
sourceFolderUri = folderUriString.toUri(),
metadata = metadata
)
}
}
suspend fun deleteFilesBySourceFolder(folderUriString: String) = withContext(Dispatchers.IO) {
recentFileDao.deleteFilesBySourceFolder(folderUriString)
}
suspend fun updateEpubReadingPosition(uriString: String, locator: Locator, cfiForWebView: String?, progress: Float) = withContext(Dispatchers.IO) {
val item = recentFileDao.getFileByUri(uriString)
if (item != null) {
@ -126,6 +168,10 @@ class RecentFilesRepository(context: Context) {
}
}
suspend fun getFolderBooksWithoutCovers(): List<RecentFileItem> = withContext(Dispatchers.IO) {
return@withContext recentFileDao.getFolderBooksWithoutCovers().map { it.toRecentFileItem() }
}
suspend fun updateBookmarks(bookId: String, bookmarksJson: String) = withContext(Dispatchers.IO) {
val currentTime = System.currentTimeMillis()
recentFileDao.updateBookmarks(bookId, bookmarksJson, currentTime)
@ -171,7 +217,11 @@ class RecentFilesRepository(context: Context) {
Timber.d("DeleteDebug: DAO - Permanently deleting ${itemsToRemove.size} files.")
itemsToRemove.forEach { item ->
item.coverImagePath?.let { deleteCachedCover(it) }
item.uriString?.let { bookImporter.deleteBookByUriString(it) }
try {
item.uriString?.let { bookImporter.deleteBookByUriString(it) }
} catch (e: Exception) {
Timber.w("DeleteDebug: Physical file deletion failed (likely already gone) for ${item.bookId}: ${e.message}")
}
}
recentFileDao.deleteFilePermanently(itemsToRemove.map { it.bookId })
Timber.d("Permanently removed recent files from DB.")